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Reasoning  and  Comprehension  Processes  of  Linguistic  Minority  Persons 

Learning  from  Text 

The  general  goals  of  the  three  year  project  "Reasoning  and  Comprehension  of 
Linguistic  Minority  Persons  learning  from  Text"  have  been  to  examine  the  processes  and 
proficiencies  called  upon  when  nonnative  speakers  of  English  attempt  to  learn  new 
information  from  written  text.  A  particular  focus  has  been  on  text  and  task  characteristics 
that  interact  with  learner  skills,  leading  in  some  cases  to  highly  successful  performance  but 
in  others  to  quite  marginal  performance.  A  specific  concern  was  to  identify  the  nature  of  the 
problems,  if  any,  that  are  unique  to  nonnative  English  speakers.  The  empiricai  activities 
conducted  under  the  auspices  of  this  project  include: 

•  A  verbal  protocol  investigation  of  the  strategies  students  used  to  answer  questions 
from  texts  typical  of  the  introductory  textbook  material  prevalent  in  first  and  second  year 
college-level  courses  (Goldman  &  Duran,  1987;  1988) 

•  An  analysis  of  the  rhetorical  and  stylistic  mechanisms  authors  of  text  use  to 
provide  cues  to  both  global  and  local  organization  of  the  presented  information  (Duran, 
Goldman  &  Smith,  1989). 

•  A  series  of  experimental  studies  of  the  effects  of  enumeration  markers  on  the 
reading  time  and  recall  behavior  of  native  and  nonnative  English  speaking  college  students 
(Goldman,  1988a,b,c). 

•  A  series  of  experimental  studies  of  native  and  nonnative  English  speaking  college 
students'  understandings  of  the  functions  and  appropriate  usage  of  four  types  of 
conjunctions,  a  category  of  cohesion  devices  that  specify  the  logical  relationships  among 
informational  units  in  text  (Goldman  &  Murray,  1989). 

Several  Macintosh  software  applications  were  developed  to  conduct  the 


experimental  studies.  Each  program  presents  text  and  allows  the  tracking  of  the  reading 
behavior  of  the  students.  One  application,  READ1TI.  presents  text  segments  in  sequential 
order  and  permits  the  student  to  go  back  and  forth  among  segments  (Saul,  Pohl,  &  Goldman, 
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1988).  Inspection  time  data  are  collected  and  there  is  an  application  that  summarizes  the 
data  for  each  passage.  The  second  application  is  a  variant  of  the  moving  window  technique 
(Gontier,  Saul,  &  Goldman,  1989).  An  entire  text  is  presented  but  the  actual  text  is  masked 
with  black  lines.  Students  select  the  sentence  they  want  to  read  and  only  that  sentence  is 
exposed.  Sequence  and  inspection  time  are  recorded.  These  applications  have  permitted  the 
analysis  and  comparison  of  reading  strategies  employed  by  native  and  nonnative  English 
speakers  as  they  perform  the  experimental  tasks. 

This  final  report  summarizes  the  results  of  the  foregoing  project  activities.  The 
overarching  theoretical  framework  and  assumptions  about  comprehension  and  reasoning  that 
have  guided  all  of  the  work  are  reviewed  first.  This  section  is  followed  by  a  discussion  of  the 
specific  investigations  and  the  major  results  of  each.  Further  information  regarding  each 
project  is  available  in  the  indicated  technical  reports  and  publications. 

I.  THEORETICAL  FRAMEWORK  OF  THE  PROJECT 

The  theoretical  framework  that  has  guided  this  work  is  that  reasoning  and 
comprehension  are  the  processes  involved  in  acquiring  new  information  from  written  text. 
We  assume  that  comprehension  is  an  interactive  process;  as  such,  the  characteristics  of  the 
learner,  the  text,  and  the  task  impact  the  cognitive  and  metacognitive  activities  associated 
with  learning  (e.g.,  Brown,  Campione,  &  Day,  1981).  A  text  may  be  said  to  be  understood 
when  a  coherent  representation  of  it  has  been  constructed  by  the  reader  (e.g.,  Kintsch  &  van 
Dijk,  1978).  The  ease  with  which  such  a  representation  can  be  constructed  is  affected  by 
text  factors  such  as  vocabulary,  sentence  structures,  and  local  and  global  organizational 
devices.  It  is  also  affected  by  how  much  the  reader  already  knows  about  the  topic  of  the  text. 
Finally,  the  purpose  for  reading  and  the  specific  task  that  the  learner  is  trying  to 
accomplish  influence  the  specific  sort  of  representation  that  will  be  most  functional. 

Comprehension  difficulties  often  arise  because  the  level  of  linguistic  proficiency 
and  the  degree  of  domain-specific  knowledge  assumed  by  text  exceed  that  possessed  by  the 
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reader.  For  example,  unfamiliar  vocabulary  may  make  a  text  difficult  to  understand.  One 
purpose  of  informational  text  is  to  communicate  the  meaning  of  key  concepts  and  terms  in 
the  domain.  Thus,  text  whose  function  is  to  impart  such  new  information  is  likely  to  have 
many  unfamiliar  vocabulary  terms.  In  addition,  comprehension  difficulty  may  be 
associated  with  understanding  how  the  words  in  a  sentence  and/or  the  sentences  in  the  entire 
passage  relate  to  one  another.  The  strategies  readers  use  for  resolving  comprehension 
difficulties  often  vary  but  can  include  ignoring  words  that  are  not  understood,  rereading,  and 
consulting  an  outside  expert  (Collins  &  Smith,  1981). 

Empirical  investigations  of  comprehension  as  an  interactive  process  involving  the 
learner,  the  task  and  the  text  are  scarce  (see  for  examples  of  such  work  Ammon,  1987; 
Langer,  1985).  In  our  project  activities  we  have  been  concerned  with  two  learner 
characteristics:  (1)  the  language  proficiency  of  nonnative  English  speakers,  and  (2) 
domain-specific  prior  knowledge.  We  have  focused  on  language  proficiency  particularly  as 
it  relates  to  the  language  used  in  academic  texts.  Prior  knowledge  is  of  interest  particularly 
as  it  interacts  with  language  proficiency.  High  versus  low  knowledge  effects  on 
comprehension  and  reasoning  have  been  demonstrated  by  a  number  of  researchers  (e.g., 
Chiesi,  Spillich,  &  Voss,  1979;  Dee  Lucas  &  Larkin,  1986;  Spilich,  Vesonder,  Chiesi,  & 
Voss,  1979).  However,  little  is  known  about  the  relationship  between  general  language 
proficiencies  and  domain-specific  knowledge  as  they  affect  the  learner’s  task  of  acquiring 
new  information  from  text. 

In  our  work  we  have  focused  on  text  characteristics  that  involve  language 
structures  that  express  relationships  among  individual  units  of  information.  Such  devices 
impact  text  cohesion  and  the  learner’s  ability  to  construct  a  coherent  representation  of  the 
text.  Some  of  the  relevant  language  structures  are  most  evident  at  the  sentence  level.  These 
include,  e.g.,  conjunctions,  conditionals,  performatives,  and  quantification  terms  and 
phrases  (see  for  further  discussion  Celce-Murcia  &  Larson-Freeman,  1983;  Halliday  & 
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Hasan,  1976).  Other  relevant  language  structures  operate  at  the  level  of  an  entire  text  and 
provide  an  organizing  relationship  for  all  of  the  ideas  in  the  text.  Examples  of  such  global 
structures  are  compare/contrast,  thesis/evidence,  and  cause-effect  .  These  global 
structures  are  typically  signalled  by  rhetorical  devices  at  the  paragraph  level  and 
awareness  of  them  allows  the  learner  to  more  easily  encode  and  construct  a  coherent  mental 
representation  of  the  incoming  information  (e.  g.  Brewer,  1980;  Meyer,  Brandt,  &  Bluth, 


1980). 

The  task  characteristics  that  we  have  been  looking  at  arise  from  a  consideration  of 
comprehension  skills  hierarchies  (Bormuth,  Manning,  Carr  &  Pearson,  1970;  Rosenshine, 
1980)  and  envisionment  levels  (Fillmore,  1983;  Kay,  1987;  Langer,  1986). 
Comprehension  tasks  can  be  thought  of  as  varying  along  a  continuum  reflecting  the  degree  to 
which  the  task  can  be  successfully  completed  with  "only"  the  text  as  compared  to  requiring 
material  and  knowledge  external  to  the  text  (see  for  discussion  Goldman,  1985). 
Comprehension  skills  hierarchies  reflect  this  continuum  in  that  these  ordered  skills  imply 
an  increasingly  more  sophisticated  understanding  as  one  moves  from  "literal" 
comprehension  of  the  text,  to  making  simple  inferences  from  the  text,  to  engaging  in  more 
complex  inferential  reasoning  based  on  the  information  in  the  text.  Variations  in 
dependency  on  the  text  are  also  reflected  in  the  envisionment  levels  of  reasoning  proposed  by 
Fillmore  (1983).  Envisionment  levels  refer  to  variations  in  individuals'  understanding  of 
the  world  described  in  a  text.  These  levels  range  from  the  most  basic,  i.e.,  "Understanding 
independent  statements  in  a  text",  to  the  most  complex,  i.e.,  "Embellishing  the  text  world  in 
light  of  existing  knowledge  and  in  terms  of  possible  extensions  and  underlying  generalities". 
The  more  basic  envisionment  levels  are  more  dependent  on  the  text,  reflect  more  literal 
understanding  of  the  material  and  assessment  typically  involves  different  kinds  of 
comprehension  tasks  than  does  assessment  of  more  complex  understandings. 
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Project  activities  have  examined  a  range  of  envisionment  levels  using  question 
answering  (Goldman  &  Dur^n,  1988),  cued  recall  (Goldman,  1988a,b),  and  cloze 
completion  tasks  (Goldman  &  Murray,  1989).  Furthermore  we  have  collected  several 
dependent  measures  that  permitted  inferences  regarding  metacognitive  and  strategic  aspects 
of  performance  (Goldman,  1988c).  These  measures  included  confidence  ratings,  reading 
time,  verbal  justifications,  and  a  method  for  tracking  the  sequence  in  which  the  sentences  in 
a  text  are  examined. 

II.  ANSWERING  QUESTIONS  FROM  OCEANOGRAPHY  TEXTS 

We  began  our  investigations  of  the  interaction  of  learner,  task  and  text 
characteristics  with  a  descriptive  study  of  behavior  of  students  engaged  in  answering 
questions  based  on  material  they  had  read  in  their  textbooks,  a  task  typical  of  introductory- 
level  college  courses.  Participants  were  native-English  and  ESL  speakers  who  varied  in 
their  levels  of  expertise  in  oceanography  and  oceanography-related  domains.  Task  variables 
were  manipulated  by  including  questions  for  which  the  answers  required  students  to  engage 
in  different  "levels”  of  comprehension.  In  fact,  interactions  between  text  and  task 
characteristics  created  situations  where  questions  that  looked  "literal"  actually  required 
going  beyond  the  information  in  the  text.  For  example,  if  the  text  stated  the  answer  to  a  yes- 
no  question,  the  question  is  categorized  as  a  literal  question.  On  the  other  hand,  the  answer 
to  this  same  question  might  not  be  explicitly  stated  in  the  text,  in  which  case  the  learner 
would  have  to  engage  in  inferential  reasoning  to  answer  it.  In  the  latter  case,  correctly 
answering  would  indicate  more  complex  levels  of  comprehension  than  in  the  former  case. 
Furthermore,  different  question-answering  strategies  would  be  expected  in  the  two  cases. 

Thus,  depending  on  the  nature  of  the  text,  the  task,  and  the  relationship  between  the 
text  and  the  task,  different  strategies  for  accomplishing  the  task  may  be  called  for.  Some 
questions  and  tasks  may  be  answered  in  an  almost  "rote"  fashion  by,  literally,  matching  the 
words  in  the  question  to  the  same  words  in  the  text.  Other  questions  and  tasks  may  require 
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using  the  information  in  the  text  to  draw  conclusions  and/or  relate  the  material  to  the 
learner's  own  world  (Goldman,  1985;  Langer,  1985;  Rosenshine,  1980).  In  general,  the 
difficulty  associated  with  achieving  correct  performance  was  expected  to  increase  as 
"distance"  from  the  text  increased. 

Our  purpose  in  the  research  was  to  examine  the  strategies  learners  employed, 
given  their  levels  of  expertise  in  the  domain,  their  English  language  proficiency  and  the 
relationship  between  the  particular  question  and  the  text.  For  this  purpose,  we  employed  a 
think-aloud  protocol  methodology.  Despite  the  potential  difficulty  of  talking  and  working 
with  English  text  concurrently,  we  note  that  think-aloud  procedures  have  been  efficaciously 
used  with  ESL  students  in  previous  work  on  reading  and  reading  strategies  (e.g.  Benedetto, 
1986;  Block,  1986  a,b).  Within  task  variation  was  introduced  by  including  questions  that 
varied  along  the  text-dependency  continuum  and  hence  had  the  potential  for  eliciting 
different  strategies  from  the  students. 

Method 

Materials 

Two  sections  of  text  were  selected  from  the  book  the  subjects  were  using  in  their 
introductory  oceanography  course,  Ocean  Science  (Stowe,  1983).  The  text  sections  were 
comparable  in  terms  of  length  (approximately  1500  words  in  each  selection)  and  featured  a 
representative  sampling  of  the  types  of  concepts  students  in  the  course  have  to  deal  with, 
including  definitions  of  terms,  properties  of  geophysical  phenomena,  mathematical  relations 
and  physical  laws.  (For  further  details  of  the  materials,  refer  to  Goldman  &  Duran,  1987, 
1988.) 

Comprehension  questions  were  largely  drawn  from  the  study  guide  developed  by  the 
instructor  of  the  course,  but  in  three  cases  were  developed  by  the  authors.  Correct  answers 
to  each  question  were  provided  by  the  course  instructor.  Specific  relations  between  the 
questions  and  the  texts  were  determined  by  identifying  and  comparing  the  explicit  predicate 
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propositions  and  coherence  relations  occurring  in  each.  Based  on  this  analysis  we 
determined  that  questions  varied  in  terms  of  how  closely  they  matched  the  surface  structure 
of  the  text;  in  their  reliance  on  technical  knowledge  (especially  vocabulary),  presumably 
acquired  prior  to  taking  the  oceanography  course;  and  in  their  dependence  on  reasoning 
processes  for  correct  solution.  This  variation  in  types  of  questions  permitted  the 
examination  of  different  solution  methods,  and  their  relation  to  the  various  question  types. 
Question  types 

Five  question  types  were  identified  based  on  an  analysis  of  (1)  the  relationship 
between  the  question  and  text,  (2)  the  demands  made  on  the  knowledge  base,  and  (3) 
previous  empirical  work  on  differences  between  various  types  of  questions  (e.g.,  Bormuth, 
et  al.,  1970;  Goldman,  1985;  Langer,  1986;  Rosenshine,  1980).  Table  1  summarizes  the 
five  types  and  indicates  how  the  18  separate  questions  employed  in  the  study  were 
distributed  across  types.  In  the  first  type  of  question  a  verbatim  relationship  exists 
between  the  question  and  the  text:  The  answer  is  given  explicitly  and  there  is  a  direct  match 
between  the  question  wording  and  the  text  wording.  To  answer  this  kind  of  question,  the 
learner  merely  has  to  locate  the  appropriate  section  in  the  text  and  find  the  matching 
language. 


Insert  Table  1  about  here 


The  second  type  of  question  involves  a  paraphrase  relationship  between  the 
question  and  the  text;  The  text  explicitly  gives  the  answer  certain  vocabulary 
equivalences  and  conversions  are  made  by  the  learner.  These  conversions  frequently  depend 
on  prior  knowledge  and  apparently  assume  that  the  learner  already  knows  certain  technical 
vocabulary  (e.g.  factor,  ratio)  because  such  terms  are  not  defined  anywhere  in  the  text. 

This  vocabulary  issue  may  be  extremely  cogent  for  predicting  nonnative  English  speakers’ 
success  on  these  types  of  questions.  For  this  type  of  question,  once  the  appropriate 
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vocabulary  conversions  are  done  and  conceptual  equivalences  established,  the  relevant  text 
portion  typically  is  circumscribed  and  localized  in  one  small  area  of  the  text.  Thus, 
question  .ypes  1  and  2  tend  to  involve  locating  specific  sentences  in  the  text  and  reflect 
largely  literal  comprehension  and  local  envisionment. 

The  third  type  of  question,  verbatim  look-up  plus  comparison,  involves  reasoning 
with  information  found  in  the  text.  The  text  gives  the  necessary  information  explicitly  and 
there  is  usually  a  direct  match  between  the  question  wording  and  the  text.  Thus,  locating  the 
information  proceeds  much  like  in  type  1  questions.  However,  once  the  information  is 
"found",  it  must  then  be  compared  to  other  information.  The  fourth  type  of  question 
requires  integration  of  information  across  several  paragraphs  of  the  text.  There  may  be 
either  a  verbatim  or  paraphrase  relationship  between  the  question  and  the  text  but  the  text 
provides  the  relevant  information  in  a  number  of  paragraphs.  The  information  must  be 
coordinated  and  analyzed  to  construct  the  correct  answer.  These  types  of  questions  would 
lend  themselves  to  partially  correct  answers  if  learners  were  to  locate  only  one  relevant 
section  of  text.  This  type  of  question  thus  requires  integrating  local  knowledge. 

The  last  type  of  question  requires  reasoning,  application  and/or  computation. 
Questions  of  this  type  involve  using  a  text-provided  formula,  rule  or  relationship  to  get  the 
precisely  correct  answer.  Locating  the  formula,  rule  or  relationship  involves  a  verbatim 
or  paraphrase  match  or  look-up  process  but  answers  to  questions  of  this  type  cannot  be 
found  in  the  book  directly.  Rather,  the  learner  must  disembed  the  relevant  information  and 
apply  it  to  a  new  situation  described  in  the  question.  This  type  of  question  thus  requires 
envisionment  or  comprehension  levels  that  involve  extending  ‘.he  text  beyond  its  own 
confines  whereas  the  other  four  types  of  questions  stay  within  the  boundaries  of  the  text. 
(The  texts,  questions  and  answers  are  available  in  Goldman  &  Duran,  1987) 
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SuSiecis, 

Seven  students  volunteered  to  participate  in  the  question-answering  protocol 
study.  Six  of  the  students  were  enrolled  in  the  introductory  oceanography  course  on  campus 
and  the  seventh  (HS)  was  the  teaching  assistant  for  the  course  and  our  "a  priori”  expert  in 
the  content  domain.  Students  completed  a  background  questionnaire  dealing  with  academic 
information,  language  skills  and  study  habits.  Table  2  provides  a  summary  of  the  most 
pertinent  information  from  this  questionnaire.  The  teaching  assistant  and  three  of  the 
students  were  native  English  speakers  (MR,  LH,  and  DW).  The  other  three  students  (GL,  II, 
and  EH)  were  from  three  different  nonEnglish  language  backgrounds  but  were  relatively 
proficient  in  English.  GL  rated  his  English  language  skills  "extremely  good";  II  and  EH  each 
rated  their  skills  "good".  The  four  native  English  speakers  rated  their  own  English  language 
skills  "extremely  good."  The  language  of  instruction  during  high  school  had  been  English  for 
all  the  students.  EH  had  had  the  least  exposure  to  English,  having  entered  the  United  States 
six  years  ago,  at  which  time  she  had  her  first  contacts  with  English.  II  and  GL  were  first 
exposed  to  English  at  the  age  of  4  years.  Of  the  nonnative  English  speakers,  only  EH  reported 
using  her  native  language  daily.  GL  and  II  reported  that  they  used  their  native  languages  to 
read  newspaper  or  magazine  material,  but  rarely. 


Insert  Table  2  about  here 

All  students’  high-school  backgrounds  included  courses  in  chemistry  and/or 
biology.  MR  and  DW  were  college  seniors,  GL  a  junior,  LH  a  sophomore  and  EH  and  II  were 
freshmen  at  the  time  of  the  study  and  thus  experience  with  college-level  science  courses 
varied.  MR  had  an  extensive  background  in  physics  and  HS  in  oceanography.  GL  had  taken 
astronomy.  The  other  three  students  had  no  college-level  science  courses  but  II  had  had 
physics  in  high  school.  Based  on  this  background  information,  it  appeared  that  DW,  LH,  EH 


and  perhaps  II  would  have  less  prior  knowledge  relevant  to  questions  on  oceanography  than 
the  other  three.  Table  2  also  describes  the  self-ratings  of  English  language  skills  related  to 
academic  performance.  These  ratings  suggest  that  the  native  and  nonnative  English  speakers 
differed  primarily  in  terms  of  their  command  of  science  vocabulary  with  the  latter 
indicating  only  moderate  command  of  such  terms.  Ratings  on  learning  from  English  lectures 
were  "good"  and  "extremely  good"  for  all  students.  Learning  from  English  text  skills  were 
rated  higher  ("extremely  good”)  by  the  native  English  speakers  as  compared  to  the 
nonnative  English  speakers  ("good"). 


Volunteers  were  recruited  from  students  in  the  introductory  oceanography  class. 
They  were  oaid  $5  per  hour  for  their  participation.  Students  were  told  that  we  were 
interested  in  how  they  went  about  answering  questions  on  oceanography  material  and  that  we 
wanted  them  to  think-aloud  as  they  worked  on  several  questions  that  we  would  give  them. 
Then  the  think-aloud  method  was  described  and  modeled,  following  procedures  outlined  by 
Ericsson  and  Simon  (1984). 

Following  practice  with  the  verbal  protocol  procedure,  the  first  text  selection  was 
presented  and  subjects  were  free  to  read  over  it.  Then  the  questions  were  presented  one  at  a 
time  for  the  students  to  answer.  Interviewer  probes  were  used  to  (1)  encourage  subjects  to 
report  what  they  were  looking  at  and  thinking  about  as  they  answered  each  question,  and  (2) 
clarify  and  disambiguate  some  of  the  subjects’  comments.  Approximately  3-4  weeks  later, 
students  i-  .,med  and  they  worked  with  the  second  text  selection.  After  it  was  read  over,  the 
questici  ..  ./-re  presented  one  at  a  time  and  subjects  were  asked  to  answer  them.  Sessions 


were  a udiui;^  ?o  and  later  transcribed. 


Analysis  of  the  Question-Answering  Protocols 
Conceptual  Process  model 

We  developed  a  general  conceptual  model  of  the  processes  and  sources  of 
information  that  might  be  involved  in  answering  questions  based  on  academic  text  materials. 
This  conceptual  model,  shown  in  Table  3,  indicates  four  major  processing  events  and  the 
goals  associated  with  each  one.  In  addition,  metacognitive  processing  events  are  shown  as 
optional.  Monitoring  can  apply  to  any  of  the  processing  events  and/or  to  learners’  general 
thoughts  about  their  performance,  the  task,  the  text,  etc.  In  discussing  the  model,  examples 
from  the  protocols  are  used  illustratively.  However,  the  conceptual  model  was  not  derived 
from  the  protocols;  rather  its  development  was  influenced  by  prior  empirical  research 
illustrating  that  comprehension  and  question  answering  may  be  profitably  conceptualized  as 
constructive,  problem  solving  activities  (e.g.,  Ammon,  1987;  Bransford  &  Johnson,  1973; 
Collins,  Brown  &  Larkin,  1980;  Goldman,  1985;  McLaughlin,  1987).  We  were 
particularly  interested  in  the  relation  between  the  question  and  the  text  and  how  that 
relation  impacted,  if  at  all,  on  students’  solution  strategies. 


Insert  Table  3  about  here 


Each  major  processing  event  shown  in  Table  3  may  be  further  "unpacked,"  or 
expanded,  into  its  constituent  processes,  goals  and  procedures.  An  extended  discussion  of  the 
conceptual  model  appears  in  Goldman  and  Dur^n  (1988)  and  only  a  summary  presentation  is 
presented  here.  The  first  processing  event,  question  encoding,  will  be  used  to  illustrate  the 
nature  of  the  model.  Question  encoding  has  two  primary  goals.  The  first  goal  is  to  determine 
the  type  of  answer  required  by  the  question.  Essentially,  the  learner  needs  to  have  some 
sense  of  (1)  what  form  the  answer  will  take,  i.e.,  one  word,  a  sentence,  several  sentences; 
(2)  the  structural  requirements  on  the  answer,  i.e.,  two  concepts  must  be  compared  versus 


1  2 

discussing  a  single  concept;  (3)  whether  verbal  or  numeric  information  is  sought;  and  (4) 
the  likelihood  that  the  answer  was  already  given  in  the  text  or  is  present  in  memory,  i.e., 
does  the  question  wording  indicate  that  a  novel  application  or  situation  is  being  addressed? 

Without  even  considering  the  relation  between  the  question  and  the  text,  differences  in  these 
four  areas  imply  that  different  strategies  are  likely  to  be  useful  and  that  arriving  at  the 
answers  involves  differential  degrees  of  cognitive  effort. 

Furthermore,  as  we  discussed  in  the  presentation  of  question  types,  questions  vary 
in  terms  of  (1)  the  level  of  envisionment  demanded  (  Langer,  1985),  (2)  the  relationship 
between  the  text  and  the  question  (Pearson  &  Johnson,  1978;  Rosenshine,  1980)  and,  (3) 
the  type  and  extent  of  reasoning  necessary  to  arrive  at  the  answer  (Goldman,  1985). 

Variations  in  the  reasoning  requirements  of  questions  are  not  coterminus  with  the  explicit- 
implicit  factor,  nor  with  the  envisionment  level.  They  depend  on  the  form  of  the  target 
answer:  Some  questions  involve  simply  reporting  a  specific  "fact"  or  datum;  others  require 
explanation,  comparison  and  sometimes  application  to  a  new  situation. 

Two  sample  questions  illustrate  the  na^ire  of  the  differences  among  the  questions 
and  kinds  of  answers  demanded.  In  the  example  question  below  there  are  several  parts,  each 
of  which  requires  a  numeric  response,  but  which  differ  in  terms  of  how  that  numeric 
response  may  be  determined: 

a)  Suppose  air  at  25°  C  is  saturated  (100%  relative  humidity).  What  fraction  of  the  air  is 
water?  (b)  What  would  be  the  answer  if  the  temperature  were  35°?  (c)  15°?  (d)  5°? 

(e)  Does  the  answer  change  by  roughly  a  factor  of  2  for  every  10°? 

The  answers  to  parts  a  and  e  can  be  found  verbatim  in  the  text;  the  answers  to  c  and  d  can 
be  read  directly  from  a  table  given  in  the  text.  The  answer  to  b  must  be  computed  but  it  can 
be  computed  in  two  ways.  The  individual  can  either  apply  the  rule  referred  to  in  part  e  of 
the  question  or  can  interpolate  from  the  series  given  in  the  table  in  the  text.  A  second 
example  question  illustrates  an  explanation  question  where  vocabulary  is  a  critical  issue: 
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"Briefly  explain  how  'echoes'  can  be  used  to  measure  the  depths  of  discontinuities  in  the 

earth's  internal  structure.”  To  locate  relevant  portions  of  the  text  or  to  access  the 

appropriate  concept  in  memory,  the  learner  must  understand  the  equivalence  between  the 

phrase  in  the  question,  depths  of  discontinuities  ,  and  the  phrase  used  in  the  text  interfaces 

between  materials.  Furthermore,  there  are  several  sections  of  the  text  that  are  relevant 

and  two  factors  that  need  to  be  discussed:  time  and  speed.  Time  is  discussed  in  a  three 

sentence  section  that  is  separated  by  nine  sentences  from  the  section  that  deals  explicitly 

with  the  time  and  speed  relationship.  To  give  the  complete  answer,  the  learner  must 

integrate  across  sections  of  text  and  extract  the  pertinent  relationship. 

"The  first  item  (time)  could  be  read  from  your  seismograph,  but  there  is  no  direct  way  to 
know  the  speed  of  seismic  waves  deep  within  the  earth.  Fortunately,  this  information  may 
be  inferred  from  data,  due  to  the  other  important  property  of  waves. 

This  second  property  is...."  (speed  is  mentioned  again  3  sentences  later.) 

The  text  does  indicate  that  the  learner  ought  to  read  on 'for  the  complete  answer  but 
sensitivity  to  this  rhetorical  signal  may  be  related  to  English  language  proficiency. 

To  accomplish  the  goal  of  determining  what  type  of  reasoning  and  what  type  of 
answer  the  question  requires,  the  primary  means  is  to  rely  on  the  language  of  the  question. 

Analysis  of  the  language  draws  on  prior  experiences  with  other  academic  texts  and  a 
sensitivity  to  the  semantics  of  various  question  words,  such  as  how  many,  what  and  whv.  To 
the  knowledgeable  individual,  these  questions  words  provide  cues  to  the  appropriate  form  of 
the  answer.  For  example,  a  "how  many"  question  ought  to  indicate  the  need  for  a  specific 
quantity,  a  "why”  for  a  causal  or  logical  explanation  of  the  phenomenon  mentioned  in  the 
question.  In  addition,  learners  may  use  the  task  context  and  constraints  to  further  define 
the  nature  of  the  answer.  In  an  untimed  situation,  one  can  afford  to  be  more  discursive  than 
in  a  time-limited  one.  Furthermore,  the  space  provided  for  the  answer  often  gives  the 
learner  information  regarding  the  extent  of  the  answer,  e.g.,  a  blank  in  the  middle  of  a 
sentence  versus  a  question  followed  by  a  half  page  of  blank  space. 
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The  second  major  goal  of  question  encoding  is  to  determine  starting  points  for  a 
search  space.  A  primary  means  of  doing  this  is  to  use  technical  terms  and  keywords 
mentioned  in  the  question  as  entry  points  to  memory  and/or  the  textbook.  When  the  question 
uses  words  that  match  those  used  in  the  headings  and  subheadings  of  the  text,  learners  are 
virtually  assured  of  a  reasonably  well-defined  search  space  in  the  text.  Keyword  matching 
is  also  facilitated  by  the  use  of  boldface  or  italics  in  the  body  of  the  text.  In  defining  a  search 
space  in  memory,  the  keywords  in  the  question  behave  similarly  but  the  success  of  a  match 
will  vary  depending  on  learners’  individual  mental  representations  of  the  text  information. 
Regardless  of  whether  the  learner  is  defining  a  search  space  in  the  text  or  in  semantic 
memory,  there  is  the  possibility  that  the  words  in  the  question  will  be  seductively  appealing 
as  cues,  when  in  fact  the  question  really  requires  original  thinking  or  the  application  of 
presented  material.  Questions  of  this  sort  are  not  only  at  more  complex  envisionment 
levels  but  also  may  provide  "false"  signals  to  the  learner,  by  creating  the  impression  that 
the  answer  is  "in  the  book".  Thus,  question  encoding  has  important  outcomes  for  the  Search 
component  of  the  question-answering  model  {see  Table  3)  in  that  question  analysis  helps 
define  the  search  space  as  well  as  the  type  of  answer  for  which  the  learner  is  searching. 

The  Search  processing  events  are  "unpacked"  in  a  manner  similar  to  question 
encoding.  Search  may  proceed  either  in  memory  or  in  an  external  source  such  as  a 
textbook,  notes  or  supplementary  reference  material  and  the  goals  differ  somewhat  for 
memory  compared  to  external  searches.  The  goals  for  memory  search  distinguish  between 
two  outcomes.  In  the  first  goal,  the  answer  is  found  in  memory  and  no  external  search  is 
undertaken.  If  memory  search  is  abandoned  for  external-source  search,  a  second  goal  of  the 
memory  search  is  to  determine  alternate  sources  for  the  answer.  Thus,  a  memory  search 
may  not  produce  the  answer  but  may  further  define,  and  refine,  subsequent  search  spaces. 

External  source  search  can  thus  be  facilitated  by  a  memory  search  that  does  not  yield  the 
specific  answer  to  the  question.  The  memory  search  phase  of  the  model  is  the  one  about 


which  we  have  the  least  to  say  based  on  the  protocol  data  (see  Reder,  1987,  for  a  recent 
theoretical  and  empirical  focus  on  question  answering  from  memory). 

Goals  for  external  searches  are  similar  to  those  for  memory  in  that  we  assume  the 
operation  of  decision,  monitoring  and  evaluation  processes  during  the  course  of  the  search. 

In  searching  a  textbook,  learners  may  have  three  major  goals  that  are  interdependent  and 
interrelated.  One  goal  is  to  delimit  the  search  space  by  using  the  results  of  question 
encoding  and  memory  search  to  constrain  and  guide  textbook  processing.  This  goal  becomes 
increasingly  important  as  the  amount  of  potentially  appropriate  text  material  increases.  A 
second  goal,  finding  information  relevant  to  the  question,  will  be  achieved  with  less  effort  if 
the  search  space  can  be  appropriately  limited  but  will  be  more  difficult  if  the  search  space 
is  incorrectly  constrained. 

Given  a  section  of  text  to  search,  search  may  be  global  or  guided.  Global  searches 
begin  with  the  learner  having  only  a  vaguely  defined  search  space,  e.  g.,  "I'll  look  in  the  text 
for  that.  It  must  be  there  somewhere."  Such  searches  are  typically  characterized  by 
scanning  or  skimming  and  may  be  exhaustive  or  self-terminating.  Monitoring,  evaluation 
and  decision  processes  are  tuned  to  the  occurrence  of  concepts,  vocabulary,  or  other  text 
material  that  "matches"  the  requirements  of  the  question.  When  a  match  is  encountered,  the 
learner's  attention  becomes  focused  on  that  section  of  text  and  more  careful  examination  of 
the  text  replaces  the  skimming  behavior.  The  search  process,  in  effect,  changes  to  a  guided 
search.  Guided  search  is,  from  the  outset,  targeted  at  a  defined  search  space,  a  localized  area 
of  the  text.  Keywords  and  topics  mentioned  in  the  question  are  used  by  the  learner  to 
explicitly  identify  and  focus  on  specific  sections  of  text,  e.  g.,  "This  question  is  about 
gravity.  That's  section  two.  I'll  look  there." 

The  third  goal  becomes  active  once  question-relevant  information  is  located:  the 
information  must  be  meaningfully  processed  in  the  context  of  the  task  defined  by  the 
question.  The  learner  must  extract  the  relevant  information  in  a  form  that  suits  the  task 
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demands.  For  some  questions,  just  recognizing  the  right  material  and  reading  it  from  the 
text  is  sufficient.  For  other  types  of  questions,  summarization  may  be  called  for  and  in  still 
others,  the  learner  may  need  to  engage  in  extended  reasoning  and  knowledge  application 
processes. 

A  transcribed  natural  language  protocol  for  one  of  the  subjects,  DW,  is  shown  in 
Table  4  and  illustrates  the  interdependence  and  interrelatedness  of  the  goals  and  processing 
events  in  the  question  encoding,  memory  search  and  text  search  components  of  the  model. 

This  particular  question  required  the  explanation  of  an  everyday  occurrence:  "Explain  why 
you  feel  cold  when  you  get  out  of  the  shower".  Although  the  text  explicitly  stated  the  answer 
in  one  self-contained  paragraph,  text  section  B.2S15-18,  references  to  the  scientific 
concept  involved,  evaporation,  occur  in  the  three  paragraphs  preceding,  and  in  the  four 
paragraphs  following,  the  paragraph  containing  the  answer.  DW  first  encoded  the  question 
by  reading  it,  defined  the  topic  as  one  of  the  major  ones  in  that  section  of  text,  "latent  heat,” 
and  then  reread  the  question  (lines  1-3).  She  then  gave  an  answer  from  memory,  "because 
of  the  evaporation  process",  but  evaluated  her  answer  as  insufficient  for  the  question  (line 
5).  She  proceeded  to  do  a  text  search  (line  6),  presumably  to  gather  more  information  on 
evaporation  and  why  the  process  works  that  way.  She  began  reading  in  the  appropriate 
section,  B.2  Latent  heat,  but  two  paragraphs  below  the  one  containing  the  answer  (lines  7- 
11).  She  skipped  up  the  page  to  the  topic  sentence  of  the  paragraph  containing  the  answer 
(B.2S15).  She  then  stated  an  answer  that  paraphrased  SI 5  in  nontechnical  terms  but  was 
only  partially  correct  because  her  answer  deleted  the  details  of  molecular  movement  and  its 
relationship  to  evaporation  and  temperature  (lines  13-15),  details  necessary  to  the 
explanation.  She  evaluated  this  answer  as  missing  something  (line  16)  and  adopted  a  global 
search  strategy  of  skimming  from  the  beginning  of  Chapter  12  (lines  17-21)  until  she  got 
to  the  first  sentence  of  section  B.1,  which  uses  the  keyword  "evaporate”  (line  22)  She  read 
the  next  two  sentences  (lines  23-25)  and  then  skimmed  the  remainder  of  section  B.1, 
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•assumedly  because  the  "evaporation"  match  didn't  lead  anywhere  on  the  shower  explanation 
(lines  26-37).  She  then  read  the  first  six  sentences  under  section  B.2  (lines  38-44),  and 
recognized  this  as  the  relevant  material  (line  45).  She  re-read  part  of  it  (S3  -  S6  in  lines 
46-50)  and  then  continued  reading  about  perspiration  and  evaporation  (lines  51,  52).  DW 
then  proceeded  to  paraphrase  the  information  she  had  just  read  (lines  53-61).  However, 
she  got  tangled  up,  reread  S8  and  then  reverted  to  her  original  answer.  "Well,  I  guess  I'd 
say  evaporation,  and  leave  it  at  that." 

Insert  Table  4  about  here 

DW  started  by  defining  the  search  space  as  information  on  latent  heat;  she  then 
retrieved  an  answer  from  memory,  evaporation.  This  answer  guided  her  attempts  to  search 
the  text  for  more  information.  However,  DW  apparently  failed  to  see  the  importance  of 
continuing  with  the  paragraph  starting  with  S15  (line  12);  instead  she  engaged  in  a  global 
text  search  that  took  her  away  from  the  relevant  information.  She  never  got  back  to  this 
paragraph  but  attempted  to  conclude  her  answer  to  this  question  with  a  paraphrase  of  a  less- 
relevant  text  section  (B.2S7  &  8).  However  this  section  is  the  first  section  under  her  self- 
defined  topic,  "latent  heat,"  that  mentioned  "evaporation."  She  monitored  her  understanding 
and  the  paraphrase  as  not  terribly  direct  but  instead  of  resuming  a  search  process,  she 
retreated  back  to  the  same  answer  she  had  originally  retrieved  from  memory  and  the  one 
that  provided  the  impetus  for  the  text  search  in  the  first  place.  We  can  only  speculate  on 
why  she  "gave  up”  at  this  point. 

DW's  protocol  illustrates  the  fourth  component  of  the  question  answering  model, 

Construct  and  output  an  answer  (see  Table  3).  Two  interrelated  goals  involved  in  this 
process  are  (1)  answer  the  question  completely  and  (2)  match  the  type  of  answer  to  the 
type  of  question.  DW  was  answering  a  question  that  required  an  explanation  and  she 
indicated  by  her  self-question  "But  what  is  it  that  causes  that?"  that  her  initial  answer 
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("evaporation")  did  not  really  qualify  as  an  explanation.  Another  learner,  GL,  indicated  his 
attempt  to  meet  the  goal  of  answering  completely,  after  having  given  a  correct,  partial 
answer  to  one  of  the  questions.  He  stated:  "Ok.  I'll  read  it  again  and  to  find  out  exactly  what 
these  are  used  for.  I  mean  I  know  they're  used  to  measure  depth  but  I  know  they  have  other 
(uses)."  An  optional  goal  in  the  answer  component  is  to  demonstrate  that  the  information 
has  been  incorporated  into  the  knowledge  base.  Several  of  the  students  repeatedly 
paraphrased  text  sections  they  had  just  read  aloud.  We  interpreted  this  behavior  as  their 
efforts  to  comply  with  this  optional  goal. 

A  final  processing  event  in  the  question-answering  model  is  optional  but 
encompasses  processes  related  to  the  role  of  metacog nitive  behavior  in  the  question 
answering  process.  At  any  time,  and  for  any  of  the  processing  events,  confirmation, 
monitoring  and  evaluation  may  be  invoked  by  the  learner.  The  outcomes  of  such  monitoring, 
in  part,  determine  the  sequencing  and  interplay  between  the  various  processing  events.  For 
example,  after  reading  a  lengthy  section  of  the  text,  a  number  of  students  were  observed  to 
reread  the  question.  There  are  several  related  explanations  for  why  that  particular 
sequence  occurred:  first,  the  learner  might  have  monitored  memory  for  the  specifics  of  the 
question  and  determined  that  the  trace  was  not  sufficiently  active;  second,  the  juxtaposition 
of  the  text  and  the  question  might  help  in  the  determination  of  the  appropriateness  of  the 
just-read  text  to  the  exact  question.  Reder  (1987)  has  suggested  a  third  possibility: 
rereading  may  increase  the  familiarity  of  the  concepts  and  thus  the  likelihood  of  success  for 
direct  retrieval  of  the  answer. 

Solution  strategies. 

Based  on  the  conceptual  model  of  question  answering  we  derived  a  typology  of 


solution  strategies  by  considering  which  processing  events,  goals  within  processing  events, 
and  monitoring  processes  were  involved  in  solution.  A  specific  solution  strategy  consists  of 
question  encoding,  some  type  of  search,  some  set  of  additional  processing  activities 
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(including  the  empty  set)  and  output  of  the  answer.  Four  types  of  search  were  specified, 
each  of  whic.i  could  be  accompanied  by  any  of  four  "additional"  processing  activities: 
question  analysis,  reasoning/inference,  process  monitoring  or  product  monitoring. 

The  four  search  types  were  Memory  Search  (A),  Text  Search  (B),  Both  memory 
and  text  search  (C),  and  Text  Search,  Reason  Beyond  (D).  Each  search  type  could  be 
augmented  by  the  occurrence  of  one  or  more  of  four  additional  processing  activities: 

Question  analysis  refers  to  rereading  or  analyzing  the  nature  of  the  question.  Reasoning  and 
inference  refer  to  efforts  to  logically  manipulate  given  or  remembered  information. 

Process  and  product  monitoring  refer  to  metacognitive  behavior  directed  at  the  progress  of 
any  cognitive  activity  (process  monitoring)  or  at  the  results  of  search  and  retrieval 
processes  (product  monitoring). 

Protocol  Scoring. 

Each  subject's  transcribed  protocol  for  each  question  was  analyzed  to  determine 
which  of  the  solution  strategies  was  reflected  in  the  empirical  data.  To  do  this,  we  coded  the 
natural  language  descriptions  of  the  protocols  for  cognitive  actions  (e.g.,  recall,  read, 
compare,  monitor,  evaluate)  and  the  types  of  information  on  which  the  action  operated  (e.g., 
text  material,  question,  own  knowledge).  Table  5  shows  the  coded  version  of  the  natural 
language  protocol  that  was  given  in  Table  4. 

Insert  Table  5  about  here 

Based  on  the  coded  protocol,  a  solution  strategy  for  each  question  answered  by  each 
subject  was  determined.  We  adopted  the  convention  of  a  5-place  code  for  designating 
solution  strategies.  The  first  place  indicated  the  type  of  search  (A  -  D)  and  the  remaining 
four  indicated  the  presence  (1)  or  absence  (0)  of  each  of  the  processing  activities  described 
above.  Furthermore,  a  superscript  gives  the  evaluation  of  the  final  answer  produced  by  the 
learner,  with  1  indicating  correct,  2  indicating  qualitatively  correct  (but  not 


quantitatively)  for  those  questions  where  a  specific  numerical  value  was  requested,  3 
indicating  a  partially  correct  answer,  4  an  incorrect  answer,  and  5  no  answer  given. 
Applying  this  procedure  to  DW's  coded  protocol  (Table  5)  leads  to  the  designation  C.10113. 
The  search  type  is  "C"  because  DW  recalled  an  answer  from  memory  and  also  consulted  the 
textbook  prior  to  producing  her  final  answer.  Additional  processing  activities  were 
rereading  the  question  (question  analysis)  process  monitoring  ("I  know  that  there’s 
something  in  the  book  that  I'm  missing")  and  product  monitoring  ("Why  does  the  process 
work  that  way?”).  She  is  credited  with  a  partially  correct  answer  because  she  never 
explained  how  her  answer,  evaporation,  led  to  feeling  cold,  although  she  made  two 
explanatory  attempts.  Note  that  the  5-place  code  for  the  solution  strategies  does  not 
distinguish  between  single  and  multiple  occurrences  of  processing  events,  i.e.,  if  one  section 
of  text  was  read  it  was  counted  the  same  as  if  several  sections  were  read. 

The  results  of  applying  our  scoring/coding  scheme  to  each  subject's  solution  for 
each  question  was  a  distribution  of  solution  strategies  across  learners  and  types  of  questions. 
These  data  were  the  basis  of  the  subsequent  analyses. 

Results  and  Discussion 

Several  important  trends  were  reflected  in  the  solution  strategy  data.  Consistent 
with  an  interactive,  constructive  comprehension  model,  task  characteristics  and  learner 
characteristics  affected  the  nature  of  the  solution  strategies.  Task  characteristics  were 
reflected  in  the  type  of  question:  the  various  types  were  differentially  difficult  and  there 
was  evidence  of  the  use  of  different  solution  strategies  depending  on  the  type  of  question. 
Learner  characteristics  were  reflected  in  level  of  expertise  within  the  domain  and  in 
language  proficiency.  While  individuals  tended  to  be  relatively  consistent  in  their  strategic 
approach  to  the  task,  individual  differences  were  observed  and  were  related  to  both  domain 
expertise  and  language  background.  The  trends  associated  with  task  and  with  learner  are 


discussed  below. 


There  was  a  predicted  order  of  difficulty  for  the  five  types  of  questions,  with  Type 
1  expected  to  be  the  easiest  and  Type  5  the  most  difficult.  Types  2,  3,  and  4  were  predicted 
to  be  roughly  equivalent  in  difficulty.  We  also  expected,  based  on  the  notion  that  the  need  for 
various  processing  activities  depends  on  the  relation  between  the  question  and  the  text,  that 
solution  strategies  would  reflect  variations  in  task  demands  created  by  question  factors  (see 
also  Reder,  1987,  Experiment  6).  Specifically,  reasoning/inference  and  question  analysis 
were  predicted  to  occur  less  often  for  Type  1  questions  as  compared  to  the  other  types. 
Finally,  we  predicted  that  monitoring  processes  would  occur  more  often  for  questions 
requiring  the  coordination  of  information  from  multiple  paragraphs,  e.g.,  Types  3,  4,  and  5. 

Question  type  difficulty  was  measured  by  the  percent  correct  (quantitative  or 
qualitative)  answers  and  is  shown  in  the  first  row  of  Table  6.  As  predicted.  Type  1 
questions  were  the  easiest:  89%  of  the  answers  were  correct.  Type  4  questions  were  also 
relatively  easy,  71%  correct.  The  unexpected  ease  with  which  these  were  solved  is  probably 
due  to  the  verbatim  relation  that  held  between  this  sample  of  Type  4  questions  and  the  text. 

In  this  particular  task  context  the  difference  between  Types  1  and  4  was  the  amount  of  text 
that  matched  the  question,  with  Type  4  questions  matching  over  longer  segments,  thus 
requiring  integration  over  several  sections  of  the  text. 

Insert  Table  6  about  here 

Those  question  types  requiring  reasoning  were  successfully  solved  62%  (Type  3) 
and  57%  (Type  5)  of  the  time,  whereas  Type  2  questions  were  the  most  difficult,  showing 
only  39%  successful  solutions.  We  had  expected  that  this  type  of  question  might  be  more 
difficult  for  ESL  than  for  native  English  speakers  because  of  the  vocabulary  conversions 
required.  Both  groups  had  equivalent  correct  rates  (42%  for  the  ESL  and  38%  for  the 
native  English  speakers).  However,  the  two  groups  differed  in  that  the  native  English 


speakers  were  more  likely  to  have  partially  correct  answers  (31%)  than  the  ESL  speakers, 
none  of  whom  got  partial  credit  on  answers  to  questions  of  this  type.  Thus,  the  most  difficult 
question  type,  Type  2,  was  the  type  requiring  knowledge  of  vocabulary  and  conceptual 
equivalence,  primarily  for  technical,  natural  science  and  oceanography  terms. 

Question  type  and  solution  strategies 

Consistent  with  our  expectations,  learners  engaged  in  different  solution  strategies 
depending  on  the  type  of  question  they  were  attempting  to  answer.  As  shown  in  Table  6,  for 
the  Type  1  questions,  which  have  a  verbatim  relation  to  the  text,  90%  of  the  correct 
solutions  involved  a  simple  "search  plus  retrieve  answer"  strategy.  Of  these,  58%  were 
text  searches  and  32%  were  memory  searches.  Furthermore,  about  half  of  these  single 
source  searches  involved  no  other  processing  activities.  Correct  solutions  to  Type  4 
questions,  which  also  had  verbatim  relations  to  the  text,  were  simple  "search  plus  retrieve 
answer"  strategies  85%  of  the  time,  but  about  half  of  these  were  text  and  half  memory 
searches.  Not  surprisingly  reasoning  and  inference  processes  rarely  occurred  for  either 
Type  1  or  4  questions.  Solutions  to  the  two  types  of  verbatim  quest'ons  differed  principally 
in  terms  of  the  more  frequent  use  of  monitoring  in  the  Type  4  questions. 

Type  3  questions,  verbatim  "look  up"  plus  comparison,  were  similar  to  Types  1 
and  4  in  terms  of  the  search  strategies  used  in  correct  solutions  (85%  search  single 
source).  Type  3  differed  from  Types  1  and  4  in  that  54%  of  the  correct  solutions  employed 
reasoning  and  inference  processes,  a  difference  consistent  with  the  nature  of  these  questions. 
Type  3  differed  from  Type  4  but  was  similar  to  Type  1  in  that  less  than  half  of  the  correct 
solutions  (46%)  featured  monitoring,  principally  checking  the  product  or  answer  before 
stopping.  Thus,  for  these  three  types  of  questions  that  all  in  some  measure  involve 
verbatim  relations  between  text  and  question,  there  is  a  comparable  degree  of  reliance  on 
text  and  memory  search  leading  to  correct  answer  retrieval. 
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Correct  solutions  to  Type  5  questions,  requiring  use  of  a  formula  or  rule  given  in 
the  text,  are  distinguished  from  Types  1,  4  and  3  in  terms  of  the  appearance  of  the 
disconfirming  search  strategy,  i.e.,  the  search  leads  to  the  realization  that  the  answer  is  not 
"given"  explicitly  in  the  text  but  must  be  computed  (search  Type  D).  In  conjunction  with 
this  search  type,  process  and  product  monitoring  are  very  frequent  in  the  correct  solutions 
and  are  more  frequent  than  in  the  three  preceding  types  of  questions.  Similar  to  Type  3, 
reasoning  and  inference  occurred  in  63%  of  the  correct  solutions.  Solutions  to  Type  5 
questions  usually  featured  additional  processes,  a  marked  difference  from  Types  1  and  4. 

Correct  solutions  to  Type  2  questions  were  a  cross  between  Types  1  and  4  solution 
strategies.  Like  Type  1,  83%  of  the  correct  solutions  involved  searching  one  source  and 
answering;  like  Type  4,  there  was  a  high  degree  of  monitoring  (75%  of  the  solutions). 

Note  however,  that  these  data  represent  only  39%  of  the  attempted  solutions.  In  other 
words,  for  those  questions  correctly  answered,  the  successful  vocabulary  conversion 
allowed  learners  to  treat  these  questions  like  verbatim  questions.  Given  the  high  failure 
rate  on  Type  2  questions  (61%),  consideration  of  the  solution  strategies  for  both  correct 
and  incorrect  final  answers  is  informative.  Over  all  solutions,  the  most  frequent  search 
strategy  was  a  text  search  (17  of  28  solutions);  however,  59%  (10)  of  these  failed  to  lead 
to  correct  solution.  Another  five  memory  search  solutions  (63%  of  8  memory  searches 
attempted)  failed  to  lead  to  a  correct  solution.  Two  of  three  solutions  using  both  sources 
failed.  The  inability  to  map  the  language  of  the  question  onto  the  language  of  the  text  appears 
to  have  been  the  critical  obstacle  to  successful  solution.  The  fact  that  the  native  English 
speakers  were  more  likely  than  the  ESL  to  be  at  least  partially  correct  on  this  question  type 
supports  the  conclusion  that  difficulties  on  this  type  of  question  may  be  related  to  English 
language  proficiency. 

Our  findings  indicate  that  questions  requiring  more  literal  and  explicitly 
presented  text  information  are  easier  to  answer  than  those  requiring  more  global  levels  of 


envisionment.  We  confirmed  Langer's  (1986)  finding  that  monitoring  (Langer's  equivalent 
is  "puzzling")  was  more  frequent  for  the  questions  with  higher  error  rates.  In  the  present 
study,  these  were  the  questions  involving  application  of  the  information  or  vocabulary 
conversions.  These  types  of  questions  draw  most  heavily  on  English  langage  proficiency  and 
domain-related  prior  knowledge.  We  next  consider  the  relation  between  subject-matter 
expertise  and  the  solution  strategies  of  individual  learners. 

Learner  characteristics  and  solution  strategy  patterns 

The  learners  in  this  study  differed  in  terms  of  expertise  in  the  specific  subject- 
matter  domain,  oceanography,  and  in  terms  of  oceanography-related  course  work.  The 
"experts"  were  HS,  who  was  the  teaching  assistant  for  the  oceanography  course,  and  MR, 
who  had  an  extensive  hard-science  (physics)  background.  One  of  the  nonnative  English 
speakers,  GL,  had  taken  a  related  science  course  (astronomy)  and  we  dubbed  him  "high” 
knowledge.  A  second  nonnative  English  speaker,  II,  had  taken  physics  in  high  school  whereas 
the  remaining  nonnative  English  (EH)  speaker  and  the  two  native  English  speakers  (DW  and 
LH)  had  had  no  science  courses  relevant  to  oceanography.  The  characteristics  of  the  solution 
strategies  for  each  of  these  individuals  are  discussed  in  order  of  "expertise”  and  compared  in 
terms  of  the  quantifiable  characteristics,  including  percent  correct,  i’.  *  Table  7. 

Insert  Table  7  about  here 


The  observed  differences  among  the  protocols  can  be  summarized  in  the  following 
way:  The  "expert"  native  English  learners  tended  to  rely  on  memory  searches;  the  more 
successful  learner  engaged  in  optional  answer  confirmation.  Of  the  lower  knowledge  native 
English  speakers,  one  tended  to  rely  on  memory  (LH)  and  had  a  high  rate  of  incorrect 
answers.  When  she  did  search  the  text  she  found  the  correct  answers  only  for  Type  1 
(verbatim)  questions.  Finally,  DW  relied  on  the  text  to  a  greater  degree  than  the  other 
native  English  speakers.  Her  solution  strategies  were  extensive  and  quite  often  involved 
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question  analysis,  monitoring  and  reasoning.  In  general,  the  nonnative  English  speakers 
relied  on  the  text  to  a  greater  degree  than  the  native  English  speakers.  The  highest 
knowledge  and  most  successful  nonnative  English  speaker,  GL,  relied  almost  exclusively  on 
the  text  and  engaged  in  frequent  question  analysis  and  monitoring  activities.  EH  was  less 
successful  in  her  efforts  to  use  the  text  and  was  hampered  frequently  by 
vocabulary/conceptual  gaps.  Finally,  II  used  memory  and  text  searches  but  was  frequently 
hampered  by  faulty  or  inadequate  question  encoding  or  text  interpretation. 

The  foregoing  characterizations  were  based  on  the  simple  presence  or  absence  of 
the  various  processing  activities.  A  complementary  analysis  considered  the  frequency  of 
various  processing  activities.  The  frequency  data  index  the  length  of  solution.  Two 
measures  were  computed  from  the  coded  protocols:  number  of  actions  (read,  reread,  skim, 
etc.)  summed  over  all  18  questions  answered  by  each  learner,  and  number  of  metacognitive 
actions  (monitor,  evaluate,  etc.)  also  summed  over  all  18  questions  answered  by  each 
learner.  These  data  were  used  in  conjunction  with  the  data  in  Table  7  to  develop  summary 
profiles  of  the  seven  learners.  These  summaries  are  presented  in  Table  8.  Accuracy  was 
used  to  divide  the  students  into  successful  (MR  and  GL),  moderately  successful  (HS  and 
DW),  and  less  successful  learners  (II,  EH,  and  LH).  The  profiles  capture  the  relationship 
between  this  classification  and  level  of  expertise,  length  of  solution  (based  on  the  total 
number  of  cognitive  actions)  and  degree  of  monitoring  (based  on  the  total  number  of 
monitoring  actions). 


Insert  Table  8  about  here 


The  profiles  help  to  illustrate  the  complex  interactions  of  text  and  learner  that 
were  observed.  The  two  most  successful  learners  (MR  and  GL)  differed  in  the  length  of 
solution,  with  GL  engaging  in  twice  as  many  metacognitive  and  cognitive  processing 
activities  as  MR.  Whereas  GL  was  high  knowledge,  he  did  not  have  MR's  level  of  expertise 


and  may  therefore  have  relied  more  on  the  presented  information  than  did  MR.  The  two 
moderately  successful  learners,  HS,  the  content  expert,  and  DW,  a  novice  geology  student, 
replicated  the  differences  between  GL  and  MR.  Thus,  lacking  expertise  in  the  domain, 
learners  who  did  well  tended  to  extensively  process  the  written  material  in  the  context  of 
the  question  and  to  keep  careful  track  of  their  performance. 

In  contrast,  the  three  less-successful  learners  had  in  common  a  tendency  to  answer 
quickly,  i.e.  to  engage  in  relatively  short  solutions.  Comparison  of  EH  and  II  revealed 
identical  solution  success  rates  but  showed  that  metacognitve  processing  was  twice  as  likely 
for  the  lower  knowledge  learner.  Finally,  accuracy  was  lowest  for  the  low  knowledge,  native 
English  speaker,  LH;  she  engaged  in  short  solutions  and  virtually  no  metacognitive  behavior. 

The  characterizations  portrayed  in  these  summary  profiles  suggest  that  successful 
learners  may  be  sensitive  to  the  need  for  strategies  that  compensate  for  low  knowledge  in  a 
domain.  Among  the  present  learners,  the  primary  means  of  compensation  was  relatively 
lengthy  text  processing  accompanied  by  monitoring  and  evaluation  of  the  processes  and 
products  of  solution.  Nonnative  English  speakers  who  are  sensitive  to  this  compensatory 
mechanism,  and  attempt  to  use  it,  may  be  heavily  dependent  on  text  negotiating  strategies 
and  on  the  necessary  English  language  skills.  In  contrast,  nonnative  English  speakers  with 
high  or  expert  levels  of  knowledge  in  an  area  would  be  less  dependent  on  the  language  of  the 
text;  in  fact  such  expertise  might  assist  students  less  proficient  in  English  in  learning 
English  processing  skills  in  their  content  courses. 

Summary  and  Implications 

There  were  several  important  results  of  the  analyses  of  strategies  for  question 
answering.  First,  we  found  a  great  deal  of  similarity  among  the  solution  strategies  of  the 
native  and  nonnative  English  speakers,  consistent  with  previous  examinations  of 
comprehension  strategies  of  native  and  second  language  readers  (Block,  1986a,b). 

However,  one  important  difference  was  a  tendency  for  the  nonnative  English  speakers  to  rely 


more  on  the  text  than  the  native  English  speakers.  Second,  the  more  successful  learners, 
i.e.,  those  who  tended  to  get  the  correct  answers,  engaged  in  cognitive  monitoring,  especially 
of  their  answers  or  candidate  answers.  Unsuccessful  solutions  were  associated  with  largely 
unmonitored  memory  searches  and  this  was  a  particular  obstacle  for  one  of  the  "novice" 
native  speakers.  Finally,  the  task  requirements,  captured  in  the  analysis  of  the  relation 
between  question  type  and  solution  strategy,  affected  the  characteristics  of  the  solution 
strategies.  The  simplest  strategies  occurred  for  the  verbatim  questions,  which  had  the 
highest  rate  of  successful  solution.  The  hardest  question  type  was  the  one  requiring 
vocabulary  conversion  and/or  paraphrases  to  match  the  text  and  the  question. 

The  protocol  analyses  also  revealed  that  text  searches  and  comprehension 
monitoring  were  guided,  in  part,  by  rhetorical  devices  that  signal  various  organizational  and 
semantic  information  to  the  reader  (e.g.,  Lorch,  in  press;  Dur&n,  Goldman,  &  Smith, 

1989).  In  particular,  there  was  one  section  of  the  Oceanography  text  that  used  the 
sequential  markers  first  and  second.  Both  native  English  and  ESL  speakers  were  observed 
to  use  the  term  Second  to  ask  themselves  if  they  knew  what  was  first.  Other  types  of 
rhetorical  devices  mark  the  semantic  relations  among  the  different  elements  of  information, 
e.g.,  however,  in  addition,  as  a  result.  Understanding  the  semantic  relations  signalled  by 
such  rhetorical  devices  aids  the  process  of  constructing  a  coherent,  internal  representation 
of  the  text. 

We  undertook  a  series  of  experimental  studies  that  systematically  pursued  issues 
of  comprehension  and  use  of  rhetorical  devices  to  guide  text  processing  and  information 
acquisition.  Specifically,  the  work  had  two  foci,  each  of  which  was  concerned  with  potential 
differences  between  native  and  nonnative  English  speakers.  The  first  focus  was  on  the 
effects  of  a  specific  rhetorical  device,  sequence  markers,  on  reading  behavior  and  memory 
for  the  information  read  (Goldman,  1988a,  b,  c).  The  second  focus  was  on  students' 
understanding  of  the  usage  conditions  for  four  types  of  connectors:  additives,  causals, 
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adversatives,  and  sequentials.  We  wished  to  determine  if  the  four  types  of  connectors  were 
differentially  difficult  for  readers  to  understand  and  whether  there  were  strategic 
differences  in  how  students  performed  on  a  rational  cloze  task  where  the  deleted  words  were 
connectors  (Goldman  &  Murray,  1989). 

III.  STUDIES  OF  THE  EFFECTS  OF  SEQUENCE  MARKERS  ON  THE  READING  AND 
RECALL  BEHAVIOR  OF  NATIVE  ENGLISH  AND  ESL  STUDENTS 


Four  experiments  examined  the  effects  of  sequence  markers  on  the  reading  and 
retention  performance  of  ESL  and  native  English  speaking  college  students.  However,  prior 
to  conducting  these  studies,  we  determined  that  the  sequence  marker  manipulations  did 
create  passages  that  were  differentially  easy  to  understand  (Refer  to  Experiment  1  in 
Goldman,  1988).  Three  studies  described  here  (Experiments  2,  3,  and  4  in  Goldman, 

1989)  pursued  the  reading  and  information  retention  issues.  Experiment  2  examined  recall 
of  target  information  by  monolingual,  native  English  speaking  students.  Experiment  3 
examined  reading  behavior  and  retention  by  ESL  students  and  Experiment  4  looked  at  reading 
and  retention  by  native  English  speaking  students.  In  these  three  studies,  the  contrast  was 
between  retention  of  four  major  points  when  they  were  each  marked  by  the  presence  of  an 
enumeration  term  versus  when  they  were  not  marked.  In  an  additional  study  of  reading  and 
retention  (Experiment  5),  we  examined  the  effect  of  enumerating  some  but  not  other  major 
points  within  the  same  text. 

Materials 

The  materials  for  the  Experiment  1-4  were  identical.  Sixteen  introductory  texts 
containing  instances  of  enumerated  items  were  drawn  from  the  following  domains:  cultural 
anthropology,  geography,  sociology,  economics,  psychology,  oceanography,  political  science, 
biology,  physical  geography,  and  film  studies.  The  texts  were  then  modified  to  reflect  four 
conditions  that  varied  in  their  use  of  signalling  devices.  After  modification,  all  passages 
were  between  350  -  450  words  in  length,  contained  19  or  20  sentences,  and  ranged  in 
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difficulty  level  from  grade  10  thru  12,  with  two  at  the  grade  8  level,  according  to  the  Flesch 
Reading  Ease  Grade  equivalent  index. 

There  were  four  signalling  conditions  for  each  passage.  All  four  versions  had  the 
same  title,  began  with  the  same  three  introductory  sentences,  and  concluded  with  the  same 
three  sentences.  A  sample  text  is  shown  in  Table  9.  The  four  signalling  conditions  were  as 
follows. 

Insert  Table  9  about  here 

1.  Full  signalling  (Full):  The  topic  sentence  in  the  passage  explicitly  cued  the 
number  of  target  points  to  be  discussed.  Then  the  first  target  point  was  given  and  was 
explicitly  marked  hy  the  word  First.  This  sentence  was  followed  by  two  sentences  that 
elaborated  it.  The  next  target  point  followed  and  was  explicitly  marked  by  the  number  word 
Second.  It,  too,  was  elaborated  with  two  sentences.  Similarly  for  target  points  three  and 
four. 

2.  Topic  sentence  enumeration  cue  (Number):  The  topic  sentence  explicitly  cued 
the  number  of  target  points  to  be  discussed.  However,  within  the  body  of  the  text,  the 
individual  points  were  not  explicitly  marked,  i.e.,  the  words  first,  second,  third  and  fourth 
did  not  occur  in  the  text.  Otherwise  the  text  in  this  condition  was  identical  to  the  Full 
version. 

3.  Vague  enumeration  cue  (Vague):  This  condition  was  identical  to  condition  2 
except  that  in  the  topic  sentence  a  vague  quantity  term  was  used  instead  of  the  specific 
number,  e.g.,  several  appeared  in  place  of  the  word  four  in  the  topic  sentence. 

4.  No  signalling  (None):  In  the  fourth  condition,  the  topic  sentence  contained  no 
reference  to  the  number  of  points  to  be  discussed.  The  body  of  the  text  was  identical  to  that 
appearing  in  the  Number  and  Vague  conditions. 
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Experiment  1  involved  the  collection  of  normative,  rating  data  on  the  passages.  The 
results  of  this  study,  which  is  fully  described  in  Goldman  (1988b),  indicated  that  idea 
development  was  easier  to  follow  in  the  Full  signaling  passages,  and  that  they  were  more 
comprehensible  than  passages  in  the  other  three  conditions.  On  the  other  hand,  ratings  of 
ease  of  understanding  the  vocabulary  did  not  differ,  consistent  with  our  hypotheses  about  the 
locus  of  sequence  marker  effects. 

Experiment  2:  Effects  of  Sequence  Markers  on  Recall  by  Native  English 

Speaking  Students 

The  purpose  of  Experiment  2  was  to  examine  the  effects  of  signalling  on 
information  retention.  Retention  was  measured  by  recall  of  the  target  points  in  the  passages 
in  response  to  a  probe-sentence  cue.  It  was  expected  that  recall  of  target  information  would 
be  greatest  for  texts  presented  in  the  Full  signalling  condition;  that  the  Number  topic 
sentence  condition  would  be  superior  to  the  Vague;  and,  that  each  of  these  would  be  better 
than  None,  in  which  there  was  no  signalling  of  the  target  points. 

Method 

Subjects 

Thirty-two  introductory  Psychology  students  participated  in  the  experiment  as 
part  of  the  course  requirement.  All  were  native  English  speakers. 

Materials,  Design,  and  Procedure 

The  16  passages  used  in  Experiment  1  were  divided  into  two  sets  using  a  matched 
pairs  procedure  and  the  data  from  Experiment  1 .  Four  versions  of  each  passage  were 
assigned  to  four  different  booklets  using  a  Latin  Square  counterbalancing  procedure  to 
insure  that  each  booklet  contained  two  instances  of  each  condition.  Across  booklets  and 
across  subjects  each  passage  occurred  in  all  four  conditions  an  equal  number  of  times  and 
each  subject  received  two  different  passages  in  each  condition.  The  design  was  a  mixed 
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design  with  set  the  between-subjects  factor  (2  levels)  and  signalling  condition  the  within- 
subjects  factor  (4  levels). 

In  addition  to  the  eight  experimental  passages,  each  booklet  contained  one  practice 
passage  that  contained  no  signalling.  Subjects  were  told  to  read  the  passages  carefully 
because  they  would  later  be  asked  to  recall  the  information.  The  task  was  illustrated  with 
the  practice  passage.  The  topic  sentences  from  the  None  condition  were  used  to  cue  recall  of 
the  target  points,  e.g.,  In  the  passage  about  the  Ideal  State,  what  were  the  basic  issues  that 
concerned  Plato?  Recall  was  done  after  every  four  experimental  passages.  Subjects  were 
run  in  small  groups  in  sessions  lasting  1 .5  to  2  hours.  Details  of  scoring  can  be  found  in 
Goldman  (1988b). 

Results  and  Discussion 

The  results  of  ANOVAs  on  target  point  recall  in  each  condition,  treating  subjects  as 
a  random  factor,  are  reported  here;  analyses  treating  passage  as  a  random  factor  are 
reported  in  Goldman  (1988b).  Treatment  condition  was  the  within-subjects  variable  and 
passage  set  was  the  between-subjects  variable.  Effects  or  interactions  that  are  not  reported 
were  nonsignificant. 

For  target  point  recall,  there  was  a  main  effect  of  signalling  condition,  £  (3,  90)  = 
10.47,  o  <  .001,  Mserror  =  .567.  The  means  for  the  signalling  condition  effect  are  shown 

in  Table  10.  Four  post-hoc  contrasts  on  the  signalling  effect  were  done  and  the  Bonferroni 
procedure  was  used  to  control  for  the  Type  I  error  rate  (for  four  contrasts  alpha  level  = 
.0125  per  contrast).  Target  point  recall  was  significantly  better  in  the  Full  signalling 
condition  (M  =  2.5  out  of  4)  than  in  the  other  three  (M.  =  1.65),  £  (1,  90)  =  30.5,  a  < 
.001.  Recall  in  the  Full  condition  was  greater  than  in  the  Number  (M  =  1-73)  condition,  £ 
(1,  90)  -  8.36,  B  <  -01. 


Insert  Table  10  about  here 


Despite  the  nonsignificant  pairwise  contrasts  comparing  Number,  Vague,  and  None 
conditions,  the  data  in  Table  10  indicate  that  these  means  were  ordered  in  the  predicted 
direction.  That  is,  recall  was  somewhat  higher  in  the  Number  condition  than  in  the  Vague 


and  None  conditions  and  the  Vague  condition  was  slightly  better  than  the  None  condition. 

Thus,  as  predicted,  target  point  retention  was  better  if  passages  contained  explicit 
enumeration  of  the  target  information  than  if  these  signalling  devices  were  absent.  The 
pattern  of  data  across  the  other  three  signalling  conditions  indicated  much  weaker  effects  of 
a  signalling  device  that  was  present  only  in  the  topic  sentence.  Experiment  2  thus 
demonstrated  the  basic  effect  of  signalling  on  retention.  In  Experiments  3  and  4  we  pursued 
effects  of  signalling  on  reading  behavior. 

Experiment  3:  Effects  of  Sequence  Markers  on  Reading  and  Recall  Behavior 

of  ESL  Students 

The  purpose  of  this  study  v  as  to  examine  the  effects  of  sequence  markers  on  the 
recall  and  the  reading  behavior  of  ESL  students.  The  experiment  was  identical  in  design  to 
Experiment  2  except  that  students  read  the  material  one  sentence  at  a  time  on  a 
microcomputer  screen.  They  were  permitted  to  "flip"  backwards  and  forwards  through  the 
text.  We  recorded  this  sequence  of  behaviors.  Also,  the  time  spent  reading  (viewing)  each 
of  the  segments  comprising  the  complete  text  was  recorded. 


Method 


Subjects 


Sixteen  ESL  students  were  paid  $5  per  hour  to  participate  in  the  study.  They  were 
recruited  from  ESL  courses  at  a  university  in  Southern  California.  The  courses  from  which 
the  students  were  recruited  are  at  the  intermediate  level  of  English  language  skills  and 
students  represented  a  range  of  language  backgrounds.  For  additional  information  on  the 
characteristics  of  the  students,  refer  to  Goldman  (1988b). 


The  passages  were  the  same  as  those  used  in  Experiment  2.  They  were  again  divided 
into  two  sets,  each  set  containing  8  passages.  Thus,  there  was  one  between-subjects  factor 
(passage  set)  and  signalling  condition  (4  levels)  was  the  within-subjects  factor.  Dependent 
measures  were  number  of  target  points  recalled,  reading  rate  per  word  and  processing  time 
per  word.  The  rate  and  processing  measures  were  computed  over  the  entire  passage  as  well 
as  over  the  specific  target  point  sentences. 

Students  viewed  each  of  8  passages,  one  sentence  at  a  time,  on  a  Macintosh  Plus 
microcomputer  screen.  Specially  developed  software,  Readit!.  recorded  the  amount  of  time 
spent  on  each  sentence  and  tracked  the  subjects  forward  and  backward  movement  through  the 
sentences  in  the  text  (Saul,  Pohl,  &  Goldman,  1988).  As  in  Experiment  2,  recall  followed 
four  texts  and  was  cued  by  the  topic  sentence  from  the  None  condition.  Students  were  tested 
individually  in  sessions  lasting  approximately  2  hours. 

Results  and  Discussion 

Recall 

ANOVA  on  the  mean  number  of  target  points  recalled  (max  =  4)  indicated  a 
significant  effect  of  condition,  £  (3,  42)  =  8.14,  q,  <  .001,  Mserror  =  .447.  The  means  for 

the  condition  effect  are  shown  in  the  middle  panel  of  Table  10.  Post-hoc  contrasts  indicated 
that  Full  signalling  led  to  greater  recall  than  the  other  three  conditions,  £  (1,  42)  = 

13.87,  £  <  .01.  Furthermore,  Number  lead  to  greater  recall  than  Vague,  £  (1,  42)  =  6.32, 
U  <  .0125.  The  Full  and  Number  conditions  did  not  differ. 

Reading-time 

ANOVA  on  the  mean  rate  per  word  measure  indicated  that  there  were  no  significant 
differences  in  rate  across  the  four  conditions,  £  <  1 .  Subjects  read  at  a  rate  of 
approximately  600  milliseconds  per  word  and  all  means  were  within  20  milliseconds  of  one 
another.  The  data  are  displayed  in  Figure  1 .  For  the  mean  processing  time  measure,  also 
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shown  in  Figure  1 ,  there  was  again  no  effect  of  condition,  £  (3,  45)  =  1 .24.  Subjects  spent 
an  average  of  almost  one  second  per  word.  A  second  set  of  analyses  specifically  compared  the 
processing  time  per  word  in  the  target  point  sentences  when  they  were  signalled  and  when 
they  were  not.  In  the  Full  condition,  the  mean  process  time  per  word  was  1 .25  and  in  the 
None  condition  it  was  1.06  seconds;  this  was  a  nonsignificant  difference  £  (1,  15)  =  3.05, 
C  =  .1 .  Despite  the  lack  of  statistically  significant  differences  on  the  processing  time 
measure,  the  magnitudes  of  the  means  are  in  the  predicted  direction  and  are  consistent  with 
the  pattern  of  recall  effects:  Students  spent  the  longest  on  passages  with  Full  signalling  and 
spent  the  shortest  amount  of  time  on  the  passages  with  Vague  or  No  signalling. 

Insert  Figure  1  about  here 

Thus,  predictions  regarding  the  effects  of  explicitness  of  marking  on  recall  of  the 
target  points  were  confirmed  for  the  ESL  students.  When  the  specific  number  of  points  was 
cued  (Full  and  Number  conditions)  recall  was  better  than  when  vague  or  no  information  was 
given  regarding  the  number  of  points  to  be  developed  in  the  text.  Predictions  regarding 
reading  behavior  were  only  weakly  supported  in  that  the  reading  rate  data  indicated  no 
differences  among  the  four  conditions.  The  processing  time  data  suggested  a  tendency  for 
students  to  spend  more  time  on  the  passages  with  explicit  signalling,  and  specifically  on  the 
target  points. 

Experiment  4:  Effects  of  Sequence  Markers  on  Reading  and  Recall  Behavior 

of  Native  English  Speaking  Students 
This  study  was  an  exact  replication  of  Experiment  3  except  that  monolingual, 
native  English  speaking  students  served  as  subjects. 


Method 


Subjects 

Sixteen  native  English-speaking  students  enrolled  in  an  introductory  Psychology 
course  participated  in  the  study.  Instead  of  monetary  payment,  these  students  received 
credit  toward  their  course  requirements. 

Materials.  Design,  and.  Procedure 

The  passages,  design,  procedures  and  scoring  were  identical  to  Experiment  3. 

Results  and  Discussion 

Recall 

ANOVA  on  the  mean  number  of  target  points  recalled  (max  =  4)  indicated  a 
significant  effect  of  condition,  £  (3,  42)  =  6.38,  q.  <  .01 ,  Mserror  =  .358.  The  condition 

means  are  shown  in  the  third  panel  of  Table  10.  Post-hoc  contrasts  indicated  that  Full 
signalling  (M  =  3.13)  lead  to  greater  recall  thar  'he  other  three  conditions;  £  (1,  42)  = 
17.31,  u  <  .001.  The  other  three  conditions  did  not  differ  from  one  another  (M  =  2.41),  £s 
<1.  Recall  in  the  Full  condition  was  better  than  in  the  Number  condition,  £  (1,  42)  =  7.06, 
B.  <  .0125.  Thus,  as  predicted,  recall  was  better  the  more  explicit  the  signalling. 

Reading  Time 

Anova  on  the  mean  rate  per  word  indicated  a  significant  effect  of  condition,  £  (3, 
45)  =  3.18,  p.  <  .05,  Mserror  =  8.  Post-hoc  contrasts  indicated  that  in  the  Full  condition, 

significantly  more  time  was  spent  on  each  word  read  (Mn  =  468  milliseconds)  than  in  the 
other  three  conditions  (M  =  397  milliseconds),  £  (1,  45)  =  7.63,  p  <  .01.  The  Number, 
Vague  and  None  conditions  did  not  differ  from  one  another.  These  means  are  shown  in  Figure 
2.  Similarly,  ANOVA  on  process  time  per  word  indicated  a  significant  effect  of  condition,  £ 
(3,  45)  =  3.58*  p.=  .02,  Mserror  =  16.  Contrasts  on  these  means  indicated  that  in  the  Full 
condition  significantly  more  time  was  spent  on  each  word  (M  =  662  milliseconds)  than  in 
the  other  three  conditions  (M.  -  575  milliseconds),  £  (1,  45)  =  5.63,  £  <  .025.  Process 
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time  per  word  in  the  target  sentences  was  longer  when  they  were  signalled  (M  =  -834)  than 
when  they  were  not  (M.  =  -634  in  the  None  condition),  £  (1,  15)  =  5.23,  Q  =  .04.  The 
reading  rate  and  processing  time  data  indicate  trends  in  the  predicted  direction:  the  more 
explicit  the  signalling,  the  slower  the  subjects'  rates  of  reading  and  the  more  time  they 
spent  on  the  passage . 


Insert  Figure  2  about  here 


The  performance  patterns  indicate  that  the  primary  locus  of  the  effect  of  number 
signals  is  on  attentional  allocation  during  reading  for  the  monolingual  English  speakers:  in 
the  Full  condition  the  passage  was  processed  more  than  in  the  Number  condition  and  there 
was  greater  recall  in  the  Full  condition.  That  the  Number  and  Vague  conditions  did  not  differ 
significantly  suggests  that  when  the  signalling  merely  provides  a  criterion  for  recall 
completion,  retention  by  native  English  speakers  is  not  substantially  enhanced. 

Experiment  5:  Linguistic  Proficiency  and  the  Effects  of  Sequence  Markers 
on  Strategic  Reading  Behavior  and  Recall 
This  experiment  pursued  two  issues  suggested  by  our  earlier  results.  The  first  issue 
concerned  the  role  of  linguistic  proficiency  in  the  effects  of  the  markers  on  recall. 
Essentially,  we  were  interested  in  whether  there  were  differences  related  to  linguistic 
proficiency  in  the  ability  to  use  sequential  markers  to  aid  in  retrieval  of  the  signalled 
information.  Accordingly,  we  sampled  two  levels  of  linguistic  proficiency  among  nonnative 
speakers  of  English.  The  second  issue  concerned  the  more  direct  examination  of  the  role  of 
the  sequence  markers  in  comprehension  monitoring.  Within  the  same  text,  some  sentences 
were  marked  with  sequence  terms  while  others  were  not.  The  specific  predictions  were  that 
backtracking  in  the  text  would  be  related  to  disjunctures  in  the  enumeration  of  four  target 
points  in  the  text.  For  example,  in  one  condition,  the  first  and  third  points  in  a  four  point 
text  were  preceded  by  the  words  First  or  Third,  respectively.  When  the  student  reads 


"Third . "  if  they  were  monitoring  their  understanding,  they  should  have  backtracked  to 

try  and  find  the  second  point.  This  backtracking  should  occur  if  readers  use  the  enumeration 
signals  to  aid  their  construction  of  an  internal  representation  of  the  text  and  if  they  have  not 
already  coded  a  second  point.  That  is,  readers  with  sufficient  skill  may  recognize  the  second 
point  even  without  an  explicit  marker,  and  thus  absence  of  the  explicit  marker  would  not 
affect  their  reading  behavior. 

Method 

Subjects 

There  were  32  monolingual  native  English  speakers  and  a  total  of  32  ESL  speakers 
who  participated  in  this  study.  These  students  participated  as  part  of  a  requirement  in  their 
introductory  Psychology  course.  The  ESL  speakers  consisted  of  two  proficiency  levels,  with 
16  students  at  each  group.  ESL  2  consisted  of  students  who  were  taking  the  two  most 
advanced  courses  in  the  English  language  ESL  sequence.  ESL  3  consisted  of  1 6  students  who 
were  enrolled  in  the  two  lower  level  English  language  courses  in  the  ESL  sequence.  A  variety 
of  native  languages  were  represented  by  the  students  in  each  of  the  ESL  classes.  Asian 
languages  accounted  for  88%  of  the  ESL  3  and  38%  of  the  ESL2  students;  European  languages 
accounted  for  the  remainderof  the  students.  The  majority  of  students  in  all  of  the  language 
groups  were  freshmen  and  sophomores 
Materials.  Design  and  faocetiure 

The  materials  were  the  passages  used  in  Experiments  1  -  4,  modified  to  implement 
the  manipulation  of  the  location  of  the  sequence  markers.  There  were  again  four  conditions, 
but  they  differed  from  those  used  in  the  previous  experiments.  Specific  to  this  study  were 
the  following  conditions: 

1.  In  the  Number-Only  condition,  the  topic  sentence  gave  the  specific  number  of 
points  but  none  of  the  iarget  points  were  signalled  in  the  text.  This  condition  was 
comparable  to  the  Number  condition  of  Experiments  1  -  4. 
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2.  In  the  Number  -  1  &  3  condition,  the  topic  sentence  gave  the  specific  number 
of  points  and  only  the  first  target  point  and  the  third  target  point  were  signalled  by  the  word 
First  or  Third,  respectively. 

3.  In  the  Vague  - 1  &  3  condition,  the  topic  sentence  used  a  vague  numerical  term 
(e.g.,  several)  to  indicate  how  many  target  points  there  were  and  only  the  first  target  point 
and  the  third  target  point  were  signalled  by  the  word  First  or  Third,  respectively. 

4.  In  the  Vague  -  2  &  4  condition,  the  topic  sentence  was  the  same  as  in  the  Vague 
1  &  3  condition  but  only  the  second  and  fourth  target  points  were  signalled  by  the  word 
Second  or  Fourth,  respectively. 

DeOils  of  the  procedure  were  similar  to  that  described  for  Experiments  3  and  4, 
with  subjects  reading  two  passages  in  each  condition.  In  addition  to  the  variables  described 
for  those  experiments,  there  were  three  levels  of  linguistic  proficiency,  native  monolingual 
English,  ESL2,  and  ESL3.  Dependent  measures  were  number  of  target  points  recalled  and 
processing  time  for  the  entire  passage  and  for  the  specific  target  points. 

Results  and  Discussion 

Of  primary  interest  was  whether  target  points  were  recalled  better  when  they 
were  signalled  as  compared  to  when  they  were  not.  Performance  in  the  Number  -  Only 
condition  provided  "baseline"  data  on  recall  of  the  four  points  when  none  were  signalled. 

Table  11  provides  the  mean  recall  by  each  language  group  of  each  target  point  in  three 
conditions:  Unsignalled  when  no  points  were  enumerated  (Number-only  condition), 

Signalled  (enumerated  points  from  the  other  three  conditons),  and  Unsignalled  (points  that 
had  not  been  enumerated  in  the  other  three  conditions).  The  recall  data  from  the  Number- 
only  condition  were  submitted  to  a  multivariate,  repeated  measures  Anova  in  which  language 
group  (Native  English,  ESL2  and  ESL3)  was  the  between  subjects  factor  and  target  point  (1 , 

2,  3  or  4)  was  the  within-subjects  factor,  consistent  with  predictions,  the  results  of  the 
Anova  indicated  no  differences  in  recall  among  the  four  points,  £  (3,  59)  <  1.  Language 


proficiency  had  a  significant  effect  on  performance  in  this  condition,  £  (2,  61)  =  3.83,  a  < 


.03,  MSerror  =  1.012.  The  ESL2  group  (M  =  1-1)  recalled  significantly  more  than  the 
ESL3  group  (M  =  -61),  but  the  native  English  students  recalled  the  same  amount  (M  =  -91) 
as  each  of  the  ESL  groups,  Tukey-Kramer  procedure  critical  difference  =  .60.  Note  that  this 
group  of  native  English  speakers  generally  performed  more  poorly  (M  =  46%  of  the  target 
points)  than  their  counterparts  in  Experiments  4  (M  in  Number  condition  =  64%)  but  at 
about  the  same  level  as  the  students  in  Experiment  2  (M  =  43%).  The  ESL2  group 
performed  at  a  roughly  comparable  level  (55%)  in  the  present  case  and  in  Experiment  3 
(61%). 

Insert  Table  11  about  here 

For  comparisons  of  signalling  effects  within  a  passage,  we  used  the  recall  data  from 
the  Number  -1  &  3,  Vague  -1  &  3  and  Vague  -2  &  4  conditions.  In  the  passages  in  these 
conditions,  half  the  target  points  were  enumerated  and  half  were  not.  For  each  subject  we 
constructed  a  mean  for  each  target  point  when  signalled  and  a  mean  for  each  when 
unsignalled  by  averaging  appropriately  over  the  three  conditions.  For  example,  recall  for 
target  point  1 -signalled  was  the  average  of  point  1  recall  in  two  conditions:  Number  -  1  &  3 
and  Vague  -1  &  3;  target  point  1 -unsignalled  was  derived  from  recall  in  one  condition:  - 
Number  -  2  &  4.  Similarly,  the  data  on  target  point  2-signalled  were  based  on  performance 
in  the  Number  -  2  &  4  condition;  target  point  2-unsignalled  was  based  on  average  recall  in 
two  conditions:  Number  -1  &  3  and  Vague  -  1  &  3.  These  derived  data,  shown  in  Table  11, 
were  submitted  to  a  multivariate  repeated  measures  Anova  in  which  language  group  was  the 
between-subjects  and  there  were  two  within-subjects  factors:  signalled/unsignalled  and 
target  point  (1,  2,  3,  or  4).  The  only  significant  effect  was  language  group,  £  (2,  61)  = 
5.46,  c  =  .01.  Mean  recall  by  native  English  (M  =  -93)  and  by  the  ESL2  (M  =  -94)  groups 
did  not  differ  but  was  higher  than  recall  by  the  ESL3  group  (M  =  -60),  although  the 


differences  failed  to  exceed  the  Tukey-Kramer  critical  value  of  .6,  ft  <•  .05.  It  is  interesting 
to  note  that  the  patterns  reflected  in  the  means  in  Table  11  suggest  that  marking  was  more 
important  for  the  second  and  fourth  points  than  for  the  first  and  third  in  the  two  more 
English-proficient  groups  of  students.  The  processing  time  data  indicated  a  similar  pattern 
of  suggestive  but  nonsignificant  results. 

Given  these  results,  we  did  not  pursue  analyses  of  the  reading  behavior.  Although 
the  manipulation  of  signalling  instantiated  in  this  study  failed  to  show  the  predicted  effects, 
we  believe  that  with  longer  texts  and/or  more  complicated  information,  we  might  well 
observe  the  predicted  marking  effect  within  passages. 

General  Discussion  of  Sequence  Marker  Studies  2*4 
The  monolingual,  native  English  speakers  in  Experiments  2  and  4  showed  the  same 
pattern  of  effects  for  the  sequence  marker  variable:  Whether  passages  were  presented  in 
booklet  or  Computer  form,  the  Full  signalling  condition  produced  the  highest  recall  of  the 
targeted  information.  The  ESL  students'  performance  was  also  consistent  with  our 
predictions:  recalling  the  most  from  passages  presented  in  the  Full  signalling  condition. 
However,  patterns  of  significant  differences  among  the  four  conditions  appeared  to  differ 
somewhat  for  the  native  English  and  ESL  students  and  imply  differences  in  the  primary  locus 
of  the  effects  of  these  signals.  For  the  native  English  speaking  students  in  both  experiments, 
recall  was  significantly  better  in  the  Full  condition  than  in  the  Number  condition  but  the 
Number  and  Vague  conditions  did  not  differ.  At  the  same  time,  the  more  explicit  the 
signalling,  the  longer  the  native  English  speakers  spent  reading  the  passages.  This  pattern 
suggests  that  for  the  native  English  speakers,  .he  primary  locus  of  the  signalling  effect  was 
on  the  amount  of  time  they  spent  reading  the  passages;  there  was  a  weaker,  secondary  effect 
due  to  the  provision  of  a  criterion  for  recall  completion.  In  contrast,  for  the  ESL  students 
retention  in  the  Full  and  Number  conditions  did  not  differ  significantly  but  the  Number 
condition  led  to  better  recall  than  the  Vague  condition.  The  pattern  for  the  ESL  students 
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suggests  that  knowing  how  many  target  points  to  look  for  and  subsequently  recall  was  the 
primary  locus  of  the  signalling  effect  and  effects  on  attentional  allocation  during  reading 
were  secondary. 

However,  the  lack  of  a  significant  effect  on  attentional  allocation  in  the  reading  of 
the  ESL  students  and  the  presence  of  an  effect  in  the  native  English  speakers  should  not  mask 
the  fact  that  the  ESL  students  spent  almost  a  full  second  on  each  word  whereas  the  native 
English  speakers  spent  33%  to  50%  less  time  on  each  word.  The  ESL  students  may  exert 
very  high  levels  of  attention  to  aiUd  passage  they  read,  regardless  of  how  explicit  the 
signalling.  Because  they  are  reading  at  what  are  essentially  "ceiling"  levels  of  attentional 
allocation,  there  may  be  no  "room"  for  an  effect  of  signals  on  reading  time.  For  native 
English  speakers  reading  these  passages  probably  does  not  require  maximum  levels  of 
attention  and  they  can  respond  differentially  to  different  levels  of  signalling.  In  fact  native 
English  speakers  may  become  dependent  on  conventional  graphic  and  linguistic  signalling 
devices  to  the  point  where  they  pay  attention  only  to  information  that  is  marked  as 
important. 

Although  the  retention  performance  of  both  native  English  and  ESL  students  was 
better  the  more  explicit  the  signalling  in  the  passage,  the  precise  nature  of  the  effect  of 
signals  appears  to  be  different  for  the  two  groups.  For  the  ESL  students,  the  signals 
apparently  had  little  effect  on  their  reading  but  may  have  provided  them  with  a  criterion- 
cue,  i.e.,  the  number  of  points,  that  was  useful  during  encoding  and/or  retrieval.  There  are 
two  aspects  of  the  data  in  support  of  this  claim:  (1)  the  amount  of  time  spent  processing 
enumerated  points  was  not  different  from  that  spent  on  unenumerated  points,  and  (2)  when 
the  topic  sentence  gave  the  specific  number  of  points  (Full  and  Number  conditions)  recall 
was  equal  and  exceeded  recall  when  there  was  only  a  vague  or  no  indication  of  the  number  of 
points.  In  contrast,  the  signals  primarily  affected  the  reading  behavior  of  the  native  English 
speakers.  Having  a  criterion-cue  in  the  absence  of  specific  signals  to  the  points  was  not 


effective  with  these  students.  Unless  they  read  the  target  points  longer,  recall  was  equal. 
The  data  in  support  of  this  claim  are  (1)  native  English  speakers  spent  more  time 
processing  target  points  when  the  specific  points  were  enumerated  compared  to  when  they 
were  not  enumerated,  and  (2)  recall  when  the  topic  sentence  gave  the  number  of  points  but 
the  points  themselves  were  not  enumerated  was  lower  than  when  they  were  enumerated. 
Thus,  for  the  native  English  speakers  the  signals  appeared  to  foster  selective  attentional 
focus  during  reading;  the  criterion-cue  value  of  the  signals  was  weak.  On  the  other  hand, 
the  ESL  students  may  have  been  relying  on  the  criterion-cue  value  as  a  check  on  whether 
they  had  gotten  from  the  text  what  they  should  have.  They  read  the  text  nonselectively  until 
they  had  four  points  from  the  passage.  Regardless  of  the  different  foci  of  the  signalling 
effects,  the  present  research  indicates  that  the  presence  of  enumeration  signals  improves 
the  retention  of  the  marked  information  for  both  native  English  speakers  and  speakers  of 
English  as  a  Second  Language. 

Analyses  of  the  Reading  Strategies  of  Native  English  and  ESL  Speakers 
The  procedures  used  in  Experiments  3  &  4,  yielded  a  trace  of  the  sequence  in  which 
the  passage  sentences  had  been  read  by  each  student.  We  used  these  traces  to  examine  the 
strategies  students  were  using  when  they  read  the  sequence  marker  passages.  We  were 
particularly  interested  in  wMether  the  strategies  used  by  native  English  speakers  were 
different  from  those  used  by  the  ESL  speakers.  From  the  traces  created  by  the  Read  it  i 
program,  we  characterized  the  reading  strategies  that  students  were  using.  Initially,  we 
undertook  these  analyses  expecting  that  there  would  be  differences  in  strategy  related  to  the 
manipulations  of  the  sequence  marking  in  the  passages.  This  turned  out  not  to  be  the  case  for 
the  most  part.  Furthermore,  native  and  nonnative  English  speakers  were  far  more  similar 
in  their  strategy  profiles  than  they  were  different.  However,  several  interesting  findings 
emerged  regarding  the  identification  of  reading  strategies,  variability  within  individuals  and 
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between-group  comparisons.  (For  a  more  extended  discussions  of  this  work,  refer  to 
Goldman,  (1988c)). 

From  the  traces  of  students'  reading  behavior,  we  identified  three  types  of  global 
strategies  and  a  variety  of  local  strategies.  The  global  strategy  Once  Through ,  shown  in 
Figure  3,  is  a  straightforward,  sequential  reading  of  each  sentence  in  the  text  from  start  to 
finish.  The  other  two  global  strategies  each  involve  going  back  and  rereading  various 
sentences  in  the  text.  The  Review  Strategy,  shown  in  Figure  4,  involves  a  straight  through 
read  of  the  passage  to  the  end  (as  in  Once  Through),  followed  by  rereading  of  some  or  all  of 
the  passage.  The  third  one,  the  Regress  Strategy,  (Figure  5)  involves  backtracking  prior  to 
reaching  the  end  of  the  passage  on  the  first  pass  through  it.  After  rereading  the  end  of  the 
passage,  there  might  be  backtracking  and  rereading  of  additional  parts  of  the  passage. 

Insert  Figures  3,  4,  5  about  here 

The  eight  passages  of  each  of  the  subjects  in  Experiments  3  and  4  were  classified  as 
one  of  the  three  types  of  global  stragegies  and  proportions  for  each  subject  of  each  strategy 
type  were  computed.  With  respect  to  the  issue  of  whether  different  strategies  were 
associated  with  the  different  signalling  conditions,  there  was  no  effect  of  condition.  That  is, 
of  the  total  number  of  passages  that  were  read  with  a  Once  Through  strategy,  they  were 
spread  evenly  across  the  conditions;  similarly  for  Review  and  Regress.  This  was  true  for 
both  ESL  and  native  English  readers.  Figure  6  shows  the  mean  proportion  of  passages  in 
each  strategy  type  for  each  language  group.  Both  ESL  and  Monolingual  English  students  most 
frequently  used  the  Regress  strategy-  -  backtracking  during  initial  reading  of  the  passage. 

The  ESL  students  were  somewhat  more  likely  to  use  the  Review  strategy;  the  native  English 
the  Once  Through  strategy.  However,  all  but  one  student  used  more  than  one  type  of 
approach  and  about  a  third  of  the  students  used  all  three  strategies. 
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Insert  Figure  6  about  here 


Thus,  the  global  strategy  data  indicate  that  all  the  students  varied  the  way  in  which 
they  read  the  texts.  But  this  variation  was  not  related  to  the  structural  manipulation  that 
we  had  done:  the  Regress  Strategy  was  just  as  likely  for  Full  signalling  passages  as  for  No 
signalling  passages.  We  pursued  the  reading  strategies  further  for  two  reasons:  (1)  to 
determine  if  signals  affected  reading  behavior  at  a  more  micro,  sentence-to-sentence  level, 
and  (2)  to  determine  what  sorts  of  structural  and  semantic/conceptual  characteristics  of 
the  passage  were  guiding  the  reading  behavior. 

To  examine  the  more  micro  level,  we  identified  ten  local  or  backtracking 
strategies,  as  shown  in  Figure  7.  These  local  strategies  are  ways  of  describing  the  reader’s 
movement  through  the  text  on  a  sentence  by  sentence  level  and  capture  patterns  of  forward 
and  backward  movement  through  the  text.  The  ten  represent  two  dimensions  of  reading 
behavior:  whether  people  read  or  skimmed  and  how  much  of  the  text  they  covered  in  the 
process  of  backtracking.  In  this  figure,  heavier  lines  indicate  reading,  lighter  lines  indicate 
skimming.  Closed  circles  represent  reading  a  single  sentence. 


Insert  Figure  7  about  here 


The  first  four  strategies  (A,  B,  C,  &  D)  reflect  movement  through  the  text  in  one 
direction  over  a  relatively  long  string  of  sentences.  Strategies  A  and  B  involve  sequential 
movement  through  the  passage  over  at  least  15  consecutive  sentences:  A  involves  reading 
those  sentences,  B  skimming.  Strategy  A  implies  a  general  desire  to  reread  most  of  the  text, 
either  from  beginning  to  end  or  vice  versa.  Strategy  B  was  typically  used  to  return  to  the 
beginning  of  the  passage  prior  to  going  through  it  again  from  the  beginning.  Strategies  C  and 
D  reflect  the  use  of  both  reading  and  skimming  over  segments  at  least  12  sentences  long.  In 
C,  readers  skim  one  or  more  sentences,  read  a  sentence,  continue  skimming  one  or  more 


sentences,  then  read  a  sentence.  In  D,  readers  skim  one  or  more  sentences,  but  read  several 
sentences,  before  they  resume  skimming. 


The  other  six  strategies  reflect  backtracking  in  which  readers  reverse  the 
direction  of  their  progress  through  the  text,  using  a  combination  of  reading  and  skimming 
and  covering  relatively  short  runs  of  sequentially  ordered  sentences  (2  to  11).  Of 
particular  interest  is  Strategy  E,  which  we  refer  to  as  a  local  rereading  strategy:  the  reader 
is  reading  along  but  goes  back  and  rereads  the  prior  sentence  before  continuing  to  read.  Use 
of  E  suggests  that  readers  need  clarification  of  a  just-prior  sentence  before  going  on  in  the 
passage. 

The  frequency  distributions  of  the  local  strategies  were  similar  for  the  two 
language  groups  and  signalling  conditions.  However,  the  distributions  varied  with  the  type 
of  global  strategy.  Figure  8  shows  the  distribution  for  the  Review  Strategy.  The  four  single 
direction  strategies  -  A,  B,  C,  &  D  -  accounted  for  80%  of  the  local  strategies,  indicating 
that  backtracking  behaviors  that  occurred  after  reading  the  passage  through  once  tended  to 
involve  returning  to  the  beginning  or  near  beginning  of  the  passage  (data  for  B)  and  going 
through  the  passage  again  either  by  reading  each  sentence  (data  for  A),  or  by  selecting 
certain  groups  of  sentences  to  read  (data  for  D).  Thus,  even  after  getting  through  the 
passage  once,  additional  inspection  of  the  text  did  not  tend  to  involve  local  rereading 
strategies  that  interrupted  movement  in  a  single  direction  (none  greater  than  10%). 


Insert  Figures  8  &  9  about  here 


In  contrast,  the  local  strategies  used  in  reading  Regress  Strategy  passages  reflected 
a  relatively  high  incidence  of  backing-up-to-go-forward,  as  shown  in  Figure  9.  Prior  to 
reaching  the  end  of  the  passage,  the  local  backtracking  strategy  -  E  -  accounted  for  55%  of 
the  backtracking  for  these  passages  for  both  ESL  and  native  English  speakers.  On  these 
passages,  students  appeared  to  be  trying  to  resolve  comprehension  difficulties  and  coherence 
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relations  prior  to  taking  in  additional  new  information  from  the  passage.  The  strategy  A 
frequency  indicates  that  about  15%  of  the  time  the  local  backtracking  occurred  either  at  the 
beginning  (first  4  or  5  sentences)  or  end  of  the  text  (last  4  or  5  sentences)  and  then 
students  read  straight  through  the  remainder  of  the  text.  After  the  first  time  through  the 
passage,  the  distribution  of  local  strategies  was  quite  variable  but  generally  indicated  a 
greater  reliance  on  skimming  the  text,  with  selected  rereading  of  sentences  and  segments  of 
the  text. 

Although  local  strategy  use  was  not  specifically  related  to  signalling  condition,  it 
was  clear  that  the  traces  of  the  students  reading  behavior  were  an  excellent  vehicle  for 
examining  relationships  among  a  variety  of  text  characteristics  and  reading  behaviors 
common  to  several  local  strategies.  In  particular,  we  were  interested  in  which  sentences 
initiated  and  ended  backtracks.  Initiations  and  conclusions  of  backtracks  are  reflected  in 
direction  changes  in  the  reading  traces.  Locations  of  direction  changes  were  characterized 
with  respect  to  how  they  reflected  structural  and  semantic  characteristics  of  text. 

Structural  aspects  of  text  are  things  like  beginning,  ending,  and  explicit  rhetorical 
markers,  such  as  our  enumeration  terms.  We  were  particularly  interested  in  whether 
explicit  enumeration  of  the  target  points  was  an  important  cue  to  backtracking.  Semantic 
aspects  refer  to  various  processes  that  must  occur  to  achieve  coherence,  e.g.,  resolution  of 
anaphor,  construction  of  bridging  inferences,  etc. 

The  types  of  sentences  that  initiated  direction  changes  were  similar  for  the  ESL  and 
Native  English  speakers  but  differed  across  the  global  strategies.  For  the  Review  strategies, 
about  70%  of  the  direction  changes  were  related  to  a  structural  characteristic  of  the  text  - 
reaching  the  end  or  beginning  of  the  passage  (Figure  10).  The  topic  sentence  and  the  target 
points  accounted  for  less  than  20%  of  the  direction  changes.  However,  and  for  the  ESL 
student  only,  the  tendency  for  target  points  to  initiate  direction  changes  was  strongest  if 
they  were  explicitly  marked  (Full  condition). 
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Insert  Figures  10  &  11  about  here 


For  the  Regress  Strategy  passages  (Figure  11),  direction  changes  that  occurred 
before  getting  to  the  end  of  the  passage  were  governed  more  by  semantic  than  structural 
properties:  about  60%  of  the  target  points  initiating  direction  changes  were  in  the 
unmarked  conditions  (Number,  Vague  and  None).  This  trend  was  true  for  both  the  ESL  and 
the  native  English  speakers.  For  the  Regress  strategy  passages,  after  reaching  the  end  of  the 
passage  the  first  time,  direction  changes  appeared  to  be  related  to  more  structural 
properties  of  the  texts:  about  50%  were  initiated  by  reaching  the  beginning  or  end  of  the 
passage.  The  structural  cue  of  marking  was  important  for  the  native  English  speakers: 
explicitly  marked  target  points  accounted  for  60%  of  the  direction  changes  cued  by  the 
target  points.  This  tendency  was  not  present  in  the  ESL  students'  data. 

Thus,  the  direction  changes  were  cued  by  structural  and  semantic  properties  of  the 
texts,  depending  on  the  particular  global  strategy  that  the  students  were  using.  We  also 
found  that  both  structural  and  semantic  aspects  of  the  text  were  related  to  the  sentences  that 
students  strategically  selected  to  reread.  These  data  are  shown  in  Figure  12  for  passages 
read  with  the  Regress  Strategy.  About  65%  of  the  sentences  strategically  read  before  getting 
to  the  end  of  the  passage  the  first  time  were  in  the  introduction  or  were  target  points.  Once 
again,  explicit  marking  of  the  points  was  not  necessary  for  them  to  be  singled  out  for 
selective  rereading.  Furthermore,  the  points  elaborating  the  targets  were  strategically 
reread  almost  as  much  as  the  target  point.  Because  these  elaborations  were  not  structurally 
marked  and  are  at  the  lowest  level  in  a  content  structure,  they  imply  strategic  rereading 
guided  by  semantic  concerns. 


Insert  Figure  12  about  here 


The  strategy  analyses  indicate  that  ESL  and  native  English  speakers  have  the  same  pool  of 
available  strategies.  Furthermore,  they  appear  to  select  a  range  of  strategies  and  the 
variables  that  govern  their  selection  are  simitar.  However,  additional  research  is  needed  to 
explore  more  fully  the  processing  and  strategic  mechanisms  that  underlie  the  observed 
strategies.  Currently,  Goldman  and  Saul  (in  progress)  are  pursuing  additional  work  on 
reading  strategies  that  wi!!  enlighten  our  understanding  of  the  constraints  and  decision  rules 
that  govern  students'  selection  of  one  strategy  over  another  as  related  to  syntactic  and 
semantic  features  of  the  text. 


Sequence  markers  are  but  one  type  of  connector  used  in  writing  a  cohesive  text  or 
in  generating  a  coherent  representation  of  a  text.  Other  types  of  connectors  include 
additive  terms  that  indicate  elaboration  or  exemplification  of  already  mentioned  content, 
caudal  terms  that  indicate  cause-effect  or  antecedent-consequent  relations,  and 
adversative  terms  that  indicate  that  the  new  information  contrasts  with  or  contradicts 
previously  presented  information.  In  previous  research  with  ESL  speakers,  MacLean  and 
d'Anglejan  (1986)  examined  connectors  as  one  of  a  larger  class  of  cohesive  devices.  They 
presented  a  rational  cloze  task  to  native  English  and  ESL  speakers:  the  deleted  words  were 
various  types  of  cohesive  terms,  including  a  variety  of  connector  terms.  MacLean  and 
d'Anglejan  found  that  ESL  speakers  did  not  differ  from  native  English  speakers  in  their  use 
of  information  from  other  sentences  (across  sentence  information)  to  determine 
appropriate  completions. 

In  three  experiments,  we  used  the  rational  cloze  procedure  in  conjunction  with  a 
multiple  choice  response  task  to  examine  differences  in  difficulty  among  four  types  of 
connectors,  namely  additives,  causals,  adversatives  and  sequentials  (Goldman  &  Murray, 
1989).  The  studies  compared  the  performance  of  native  English  speakers  and  several 


groups  of  ESL  students  who  varied  in  level  of  language  proficiency  in  English.  In  the  first 
experiment,  we  compared  cloze  task  and  verbal  justification  performance  of  monolingual, 
native  English  speakers  and  ESL  university  students.  Success  in  the  university  setting  is  in 
part  determined  by  the  ability  to  learn  from  text  and  we  were  interested  in  the  levels  of 
proficiency  with  connectors  that  might  be  characteristic  of  this  population.  Based  on  the 
findings  from  Experiment  1  on  differential  difficulty  among  the  connector  types,  the 
purpose  of  Experiment  2  was  to  determine  if  monolingual  native  English  speakers’ 
confidence  in  their  answers  also  varied  with  connector  type.  Finally,  in  Experiment  3,  a 
group  of  community  college  ESL  students  who  were  at  lower  levels  of  English  language 
proficiency  than  the  university  students  completed  the  rational  cloze  task,  provided 
confidence  ratings,  and  gave  verbal  explanations  for  their  responses.  The  design  of  the  three 
experiments  can  be  understood  by  reviewing  previous  relevant  research. 

Language  Proficiency  and  Connector  Understanding 
Passages  vary  in  the  degree  to  which  the  inter-  and  intrasentential  relations 
contained  therein  are  made  explicit.  In  the  absence  of  explicit  signals  to  logical 
relationships  we  rely  on  a  "conversational  postulate"  for  text:  we  infer  cohesion  based  on  the 
default  assumption  that  successive  sentences  are  related,  making  use  of  other  cohesion 
devices  such  as  referential  and  lexical  overlap  {Halliday  &  Hasan,  1976;  Kintsch  &  van 
Dijk,  1978;  MacLean  &  D'Anglejan,1986).  Connectors,  or  conjunctives  (Halliday  & 

Hasan,  1976),  are  a  type  of  cohesion  device  that  make  explicit  the  logical  relations  among 
sentences.  Common  connectors  are  and,  but,  however,  because.  Connectors  are  of  limited 
utility  unless  the  reader  understands  how  connectors  function  in  text  and  the  logical 
relationship  each  specifies.  This  aspect  of  language  proficiency  is  particularly  important  to 
students  who  acquire  English  as  a  second  language  (ESL)  and  are  attempting  to  learn  from 
English  language  texts.  Anecdotal  reports  from  teachers  of  ESL  students  frequently  indicate 
that  learning  the  appropriate  use  of  connectors  is  an  extremely  difficult  aspect  of  the 


English  language.  To  a  lesser  extent,  this  can  also  be  said  of  native  English  speakers.  For 
example,  in  a  study  of  good  and  poor  readers,  Bridge  and  Winograd  (1982)  reported  that 
both  groups  found  it  more  difficult  to  provide  explanations  for  the  appropriateness  of 
cohesive  ties  that  they  established  with  connectors  than  those  that  they  established  with 
referential  and  lexical  cohesion  devices.  However,  good  readers  were  better  able  to  justify 
their  responses  than  were  poor  readers. 

Although  Bridge  and  Winograd  found  that  connectors  were  difficult  to  explain,  good 
readers  were  better  at  explaining  them  than  poor  readers.  There  is  also  evidence  that 
benefits  derived  from  the  explicit  inclusion  of  connectors  in  texts  vary  with  language 
proficiency.  For  example,  Geva  and  Ryan  (1985)  found  that  positive  effects  of  including 
connectors  occurred  for  skilled  and  less  skilled  readers  when  the  conjunctions  were  included 
and  highlighted  in  the  text  but  only  for  the  skilled  readers  when  the  conjunctions  were 
present  in  the  text  but  not  highlighted.  Furthermore,  Geva  and  Ryan  (1985)  found  that  the 
omission  of  connectors  negatively  affected  less-skilled  readers  but  had  no  effect  on  skilled 
readers. 

A  major  limitation  of  prior  research  on  connectors  is  the  unsystematic  and  limited 
sampling  of  connectors.  It  seems  reasonable  that  some  interclausal  and  intersentential 
relations  may  be  easier  to  infer  than  others  and  the  benefits  of  including  explicit  connectors 
would  therefore  be  differential.  In  addition,  grammar  textbooks  for  ESL  students  typically 
devote  large  sections  to  sentence  and  clause  connectors,  providing  elaborate  taxonomies  of 
connectors.  For  example,  Celce-Murcia  and  Freeman  (1983)  have  provided  a  functional 
classification  scheme  for  logical  connectors  that  differentiates  among  four  major  types  of 
connectors:  additive,  causal,  adversative,  and  sequential.  The  first  three  typically  relate  to 
interclausal  or  intersentential  relations.  Additive  relations  are  those  that  signal  some  form 
of  elaboration  of  previous  content  (e.  g.,  In  addition,  That  is).  Cause-effect  or  antecedent- 
consequent  relations  are  signalled  by  causal  connectors  (e.g.,  As  a  result,  Due  to). 


5  1 


Adversative  connectors  signal  contrastive  elaborations  (e.g.,  However,  On  the  contrary). 
Sequential  connectors  are  used  to  signal  a  more  diverse  set  of  logical  relations  and  refer  to 
the  larger  discourse  context  more  often  than  do  the  other  connectors.  Sequential  connectors 
are  used  to  enumerate  lists  of  items  (e.g.,  First,  Second,  Finally),  to  mark  a  sentence  that 
previews  the  remainder  of  the  text  (e.g.,  In  short )  or  to  indicate  temporal  sequence  of 
events  (e.g.,  Subsequently,  Later). 

Accordingly,  in  this  phase  of  the  project,  three  experiments  were  conducted  to 
investigate  two  primary  questions:  (1)  whether  the  four  types  of  logical  connectors  were 
differentially  difficult  for  readers  to  understand,  and  (2)  whether  ESL  speakers' 
understanding  of  connector  usage  was  different  from  that  of  native  English  speakers.  With 
respect  to  the  first  question,  our  expectations  were  that  sequential  connectors  would  be 
more  difficult  than  the  other  three  types  of  connectors  for  all  students  because  sequentials 
generally  require  reference  to  the  more  global  discourse  context  whereas  the  other  three  do 
not.  Among  the  other  three  connector  types,  all  indicate  some  form  of  continued  elaboration 
of  previous  information.  However,  they  differ  with  regard  to  what  they  signal  is  the 
connection  of  new  to  previous  information,  with  causal  and  adversative  connectors  signalling 
more  specific  logical  relations  than  additives.  The  latter  indicate  only  that  the  next  piece  of 
information  adds  "in  some  way"  to  the  previous  information.  In  contrast,  as  noted 
previously,  causal  and  adversative  connectors,  respectively,  signai  causal  or  logical 
contingency  or  contrast  between  old  and  new  information  in  the  clauses  joined  by  the 
connectors.  Predictions  about  the  relative  difficulty  among  these  three  connectors  were  not 
made. 

We  expected  that  ESL  speakers  would  less  frequently  supply  the  appropriate 


connector  when  compared  to  native  English  speakers.  However,  two  issues  were  of  greater 
interest  than  the  overall  level  of  performance.  If  the  four  connector  types  did  differ  in 
difficulty,  was  there  similarity  in  the  patterns  of  the  native  English  and  the  ESL  speakers? 


Second,  we  were  interested  in  the  strategies  used  to  arrive  at  connector  choices  and  whether 
these  were  similar  for  native  English  and  ESL  readers,  especially  where  differential  levels 
of  English  language  proficiency  were  present.  We  pursued  the  strategy  question  by  focusing 
on  whether  response  justifications  reflected  differences  among  the  connectors  that  are  cited 
by  text  linguistists,  discourse  analysts,  and  ESL  teachers.  We  collected  retrospective  verbal 
reports  from  students  explaining  their  choices.  We  expected  that  the  native  English 
speakers  would  be  sensitive  to  the  functional,  semantic  differences  among  the  connectors  and 
this  would  be  reflected  in  their  response  justifications.  However,  we  also  expected  ESL 
students  to  be  aware  of  usage  rules  because  they  are  specifically  taught. 

In  all  three  experiments,  subjects  completed  a  rational  cloze  task  that  had 
multiple-choice  response  options  (as  opposed  to  free  completion).  Confidence  ratings  were 
collected  in  Experiments  2  and  3.  Students  provided  explanations  of  their  responses  in 
Experiments  1  and  3.  All  native  English  speakers  were  drawn  from  the  same  population. 

The  ESL  speakers  came  from  two  populations  that  differed  in  English  language  proficiency 
levels. 

Experiment  1:  Effects  of  Connector  Types  on  the  Cloze  Completions  and 
Response  Justifications  of  Native  English  and  ESL  Speakers 

The  purpose  of  Experiment  1  was  to  compare  the  performance  of  native  English  and 
ESL  university-level  college  students  on  a  rational  cloze  task  that  required  them  to 
distinguish  among  four  logical  connectors  in  choosing  the  most  appropriate  intersentential 
connector.  The  rational  cloze  procedure  was  applied  to  passages  characteristic  of 
introductory-level  textbooks  used  in  university  settings.  Our  general  prediction  was  that 
ESL  students  would  make  fewer  appropriate  choices  than  native  English  speakers.  We 
expected  the  differential  difficulty  of  the  four  types  of  connectors  to  be  similar  in  each 
group  of  students.  Finally,  based  on  previous  research  we  expected  that  ESL  and  native 
English  speakers  would  be  similarly  aware  of  prescriptive  rules  on  the  usage  of  the  various 


connector  types  but  that  the  ESL  students  would  be  less  accurate  in  identifying  where  in  text 
the  various  rules  applied. 

Method 

Su.biec.ta- 

Participants  were  16  monolingual,  native  English  speakers  and  20  ESL  speakers, 
recruited  from  undergraduate  classes  at  a  university  in  Southern  California.  The  native 
English  speakers  were  enrolled  in  Introductory  Psychology  and  participated  as  part  of  the 
course  requirements.  The  ESL  students  were  enrolled  in  English-language  classes 
specifically  designated  for  nonnative  speakers  of  English.  Successful  completion  of  these 
classes  is  a  prerequisite  to  satisfying  the  university's  General  Education  English 
requirement.  ESL  students  who  participated  in  this  study  were  volunteers  recruited  from 
classes  at  three  levels,  but  the  majority  (15  students)  came  from  the  class  (English  1)  one 
level  below  the  university-required  English  class.  ESL  students  were  paid  $5  per  hour  for 
their  participation. 

Several  demographic  characteristics  were  assessed  using  a  questionnaire  filled  out 
at  the  beginning  of  the  experimental  session.  A  summary  of  information  most  pertinent  to 
the  present  study  is  provided  in  Table  12.  The  native  English  speakers  were  largely  college 
freshmen  and  sophomores  and  all  spoke  English  as  the  native  language.  The  mean  verbal 
Scholastic  Aptitude  Test  (SAT)  Score  of  526.8  indicates  that  these  students  were 
representative  of  the  entering  freshmen  classes  at  this  university  over  the  past  several 
years.  The  ESL  students  were  largely  freshmen  and  sophomores.  The  native  language  of 
70%  of  the  ESL  students  was  an  Asian  language  (Chinese,  Vietnamese,  Japanese  or  Korean). 
The  verbal  SAT  scores  for  the  ESL  speakers  are  typical  for  this  population.  Performance  on 
the  Test  of  English  as  a  Foreign  Language  (TOEFL)  is  the  language  proficiency  indicator 
typically  used  in  making  admissions  decisions.  The  mean  score  of  564.7  is  representative  of 
students  from  non-English  speaking  countries  who  are  admitted  to  the  university. 


Insert  Table  12  about  here 


Design  and  Materials 

The  design  was  a  mixed  factorial  with  one  between-subjects  factor  (native  English 
or  ESL  language  group)  and  one  within-subjects  factor  (connector  type).  There  were  four 
levels  of  connector  type:  additive,  causal,  adversative,  and  sequential.  Each  connector  type 
was  the  correct  response  twice  in  each  of  4  passages  for  a  total  of  8  observations  per  subject 
on  each  connector.  In  addition  to  examining  the  number  correct,  we  asked  subjects  to 
explain  their  choices  for  two  of  the  passages.  Passages  were  presented  in  four  different 
orders,  according  to  a  Latin-square  design.  Within  each  language  group",  each  presentation 
order  was  used  an  equal  number  of  times. 

Passages.  Four  passages  from  college-level,  introductory  science  and  social 
science  textbooks  were  modified  to  accommodate  2  instances  of  each  of  the  connector  types. 
The  mean  number  of  words  per  passage  was  488  and  the  mean  number  of  sentences  was 
32.75.  The  passages  were  about  biological  characteristics  of  life,  anthropological 
approaches  to  the  study  of  culture,  oceanographic  explorations  in  the  nineteenth  century, 
and  the  societal  functions  of  marriage.  For  purposes  of  illustrating  the  task  to  subjects,  an 
additional,  shorter  passage  (about  the  movie  industry)  was  developed. 

Subjects’  task  was  to  read  each  passage  and  to  choose  an  appropriate  word  to  fill  in 
each  of  the  cloze  slots  (represented  as  blanks)  that  occurred  throughout  the  passage.  Each 
cloze  slot  occurred  at  the  beginning  of  a  sentence  and  four  alternatives  were  provided.  Each 
half  of  each  passage  had  four  slots  in  it  and  each  slot  required  a  different  connector  type  as 
the  correct  answer.  A  minimum  of  one  complete  sentence  separated  successive  slots.  The 
alternatives  were  instances  of  the  four  connector  types;  the  correct  response  for  each  cloze 
slot  was  determined  by  the  experimenters.  That  the  experimenter-designated  response  was 
the  best  choice  for  the  slot  was  verified  by  a  panel  of  three  independent  judges,  one  of  whom 


is  an  ESL  specialist.  The  order  of  the  alternatives  was  systematically  varied  so  the  correct 
choice  occurred  in  each  position  (1 ,  2,  3,  or  4)  an  equal  number  of  times.  The  order  of  the 
distractors  was  also  varied  so  instances  of  the  same  connector  type  did  not  always  appear  in 
the  same  position. 

Our  classification  of  connector  types  and  the  specific  instances  used  as  response 
options  were  taken  from  the  functional  classification  scheme  proposed  by  Celce-Murcia  and 
Freeman  (1983)  and  based  on  the  earlier  work  of  Halliday  and  Hasan  (1976).  For  the 
additives  we  used  simple  additive  instances  (e.g.,  in  addition)  and  exemplification  instances 
(e..g.,  for  example)-,  for  causals,  we  used  instances  that  signal  cause/reason  (e.g.,  as  a 
result)  and  effect/result  (e.g.,  thus).  For  adversatives  we  used  instances  that  signal  that 
two  ideas  are  in  contrast  or  conflict  (e.g.,  however)  and  we  used  instances  that  signal  a 
reservation  or  restriction  in  the  applicability  of  the  preceding  information  (e.g.,  despite). 
Sequential  connectors  featured  the  largest  variety  of  subtypes  and  we  used  instances  that 
signalled  enumeration  of  points  chronologically  (e.g.,  second)  or  temporally  (e.g.,  next)  as 
well  as  those  that  indicated  summation  of  information  (e.g.,  in  short). 

The  passages  (including  the  practice  passage)  were  arranged  in  a  single  "passage" 
booklet,  and  the  alternatives  for  the  slots  for  each  passage  were  arranged  in  a  separate 
"response"  booklet.  The  order  of  the  four  passages  was  counterbalanced  such  that,  across 
all  subjects,  each  passage  appeared  in  each  position  in  the  booklet  an  equal  number  of  times. 

Procedure.  Subjects  were  run  individually  in  sessions  lasting  1.5  to  2  hours. 

Each  subject  completed  a  background  questionnaire  at  the  start  of  the  session.  After  filling 
out  the  questionnaire,  each  subject  was  instructed  on  how  to  coordinate  the  passage  and 
response  booklets  and  worked  through  the  practice  passage  during  which  time  they  could  ask 
questions  about  the  task.  For  each  blank,  subjects  circled  the  best  word  to  complete  the 
sentence  from  among  the  four  alternative  words  that  were  provided  in  the  response  booklet. 
Subjects  were  told  that  all  punctuation  associated  with  the  alternatives  had  been 


intentionally  omitted,  but  that  they  could  add  any  punctuation  they  felt  was  necessary  to 
their  response. 

Following  completion  of  the  four  passages,  the  experimenter  re-presented  the  first 
two  passages  (excluding  the  practice  passage)  the  subject  had  read.  Subjects  were  told  that 
we  were  interested  in  how  they  had  decided  on  their  answer  and,  in  why  they  thought  their 
answer  was  the  best  choice  for  the  particular  blank.  For  each  blank,  the  students  explained 
their  reasons  for  selecting  their  alternative  choices.  This  part  of  the  session  was  audiotaped 
and  later  transcribed. 

Scoring 

Each  subject  received  a  score  from  0  to  8,  indicating  the  number  of  correct 
responses  to  the  8  cloze  slots  for  each  connector  type.  We  determined  the  frequency  with 
which  each  of  the  three  distractor  alternatives  was  selected  incorrectly.  For  the  incorrect 
responses  we  also  examined  the  contingency  relation  between  the  correct  connector  and  the 
type  of  distractor  chosen.  In  the  verbal  reports,  students  tended  not  to  described  the  process 
they  went  through  in  choosing  the  answer;  rather  they  consistently  described  the  type  of 
relationship  that  held  between  the  items  of  information  that  were  being  connected.  To  score 
these  responses  a  set  of  1 1  coding  categories  was  developed.  The  categories  reflect  the 
previously  discussed  distinctions  among  the  connectors;  each  coding  category  was 
appropriate  for  only  one  of  the  connector  types.  Interrater  reliability  in  scoring  the 
responses  was  better  than  90%.  Further  details  on  the  scoring  are  available  in  Goldman  and 
Murray  (1989). 

Results 

Correct  Responses 

The  number  correct  for  each  subject  for  each  connector  type  was  subjected  to  a 
multivariate,  split-plot  ANOVA  in  which  language  group  was  the  between-subjects  factor 
and  connector  type  the  within.  The  means  for  each  connector  type  and  language  group  are 
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provided  in  Table  13.  As  expected,  the  main  effect  for  language  group  was  significant,  £  (1, 
33)  =  15.28,  £_<  .001,  Mserror  =  5.02:  The  native  English  speakers  were  correct  more 
often  (M,  =  6.34)  than  the  ESL  students  (M  =  4.88).  In  addition,  there  was  a  main  effect 
for  connector  type,  £  (3,  32)  =  3.27,  £_=  .03.  Five  post  hoc  contrasts,  using  the 
Bonferroni  procedure  (alpha  level  =  .01),  were  conducted.  Additives  and  causals  were 
correctly  answered  more  often  than  were  adversatives  and  sequentials,  £  (1,  34)  =  8.46,  £ 
•  .006,  Mserror  =  6.69.  The  other  four  contrasts  were  not  significant  nor  was  the 
interaction  between  language  group  and  connector  type,  £  (3,  32)  =  1.58. 

Insert  Table  13  about  here 

Distractor  Choices 

The  correct  response  data  indicated  that  the  ESL  speakers  made  more  errors  than 
the  native  English  speakers  -  a  mean  of  12.2  per  student  (out  of  32  possible)  compared  to 
6.62.  Regarding  these  errors,  the  following  issues  were  of  interest:  (a)  which  connectors 
made  the  best  distractors,  i.e.,  were  most  frequently  chosen,  and  (b)  whether  the 
distractors  chosen  by  the  two  groups  were  similar.  To  investigate  these  issues,  we  computed 
the  proportion  of  each  subject's  total  errors  that  reflected  each  distractor  option.  The 
proportion  data  were  subjected  to  an  arcsine  transformation  and  then  submitted  to  a 
multivariate,  split-plot  ANOVA  with  language  group  the  betwen-subjects  factor  and 
connector  type  the  within.  There  was  no  main  effect  for  language  group,  £  (1 ,  34)  .  3.04, 

£  =  .09.  However,  there  was  a  main  effect  of  connector  type,  £  (3,  31)  =  10.06,  £_<.  001 : 
Additives  and  causals  were  chosen  significantly  more  often  than  adversatives  and 
sequentials,  £  (1,  33)  =  9.48,  £  =  .004.  There  was  also  a  significant  interaction  of 
language  group  by  connector  type,  £  (3,  31)  =  4.69,  £  =  .001.  Table  14  gives  the  mean 
proportions  for  each  distractor  and  language  group.  Examination  of  the  means  suggested  that 
the  interaction  was  due  to  differences  in  the  magnitude  of  the  differences  among  distractors. 
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Accordingly,  three  difference  scores  were  computed  for  each  subject  in  each  group  and  the 
differences  between  the  distractors  were  tested  for  significance,  using  the  Dunn-Bonferroni 
procedure  to  control  the  Type  1  error  rate  (alpha  =  .017).  Causals  were  chosen 
significantly  more  than  adversatives  by  each  group  but  the  difference  was  larger  in  the 
native  English  speakers,  1  (32)  =  2.83.  The  ESL  speakers  chose  the  adversatives  and 
sequentials  equally  often  but  the  native  English  speakers  chose  the  sequentials  more  often 
than  the  adversatives,  i  (32)  =  2.46.  Although  the  means  suggest  that  the  native  English 
speakers  chose  causals  more  frequently  than  additives  while  the  ESL  speakers  chose 
additives  more  than  causals,  the  difference  between  the  groups  failed  to  reach  conventional 
levels  of  significance,  1  (32)  =  1.88. 


Insert  Tables  14  &  15  here 


An  additional  analysis  pursued  potential  dependency  relations  between  the  errors 
and  the  correct  connectors  The  presence  of  a  strong  dependency,  reflected  by  a  dominant 
distractor  for  a  particular  connector  type,  would  suggest  some  type  of  systematic 
misunderstanding  of  the  meaning  and  usage  of  that  connector.  The  proportions  shown  in 
Table  15  reflect  the  number  of  times  each  of  the  distractor  options  was  chosen  relative  to 
the  total  number  of  errors  on  each  connector.  These  dependency  matrices  indicate  a 
relatively  strong  relationship  between  additive  cloze  slots  and  causal  distractors  for  both 
the  native  English  and  the  ESL  speakers  (probability  of  choosing  a  causal  if  an  error  is  made 
on  an  additive  cloze  slot  =  .76  and  .60,  respectively).  For  each  of  the  other  connector  types, 
incorrect  responses  tended  to  be  split  between  two  distractor  options. 

Both  native  English  and  ESL  speakers  were  most  frequently  correct  when  additive 


or  causal  connectors  were  required  by  the  text;  they  also  choose  these  two  types  of 
connectors  most  often  when  they  responded  incorrectly,  and  there  was  some  evidence  of  a 
dependency  relationship  between  the  additive  cloze  slots  and  the  causal  distractor.  The 


5  9 


higher  correct  and  distractor  choice  rates  for  the  additives  and  causals  suggest  the 
possibility  that  subjects  were  generally  biased  toward  choosing  additive  and  causal  terms. 
We  examined  this  by  determining  the  rates  of  choosing  each  of  the  four  connector  types  as 
responses,  regardless  of  whether  the  response  was  correct  or  incorrect.  Then  we 
determined  the  percentage  of  choices  that  were  indeed  correct.  These  data  are  presented  in 
Table  16. 


Insert  Table  16  about  here 

According  to  the  design  of  the  passages,  each  connector  should  have  been  selected 
25%  of  the  time.  The  data  indicate  that  the  native  English  speakers  and  the  ESL  speakers 
selected  the  causals  and  the  additives  more  frequently  and  the  adversatives  and  sequentials 
less  frequently  than  a  "completely  correct"  response  profile.  However,  the  probability  of 
being  correct,  given  the  choice  of  a  particular  distractor,  was  higher  for  the  adversatives 
and  the  sequentials  than  for  the  additives  and  causals.  It  appears  that  both  native  English  and 
ESL  students  have  a  more  stringent  set  of  constraints  governing  the  appropriateness  of 
adversative  and  sequential  terms  than  those  that  govern  additive  and  causals. 

Justifications  for  Choices 

Justifications  for  Choices  that  were  Correct.  Table  17  indicates  the  distribution  of 
correct-choice  justifications  over  the  coding  categories.  The  proportions  were  computed 
separately  for  each  connector  type,  using  the  total  number  of  justifications  for  the  specific 
connector  as  the  base.  There  are  several  interesting  aspects  of  these  data.  First,  the 
distribution  of  justifications  for  the  ESL  students  is  highly  similar  to  that  of  the  native 
English  speakers:  for  each  connector  type,  the  proportional  distributions  over  the  coding 
categories  for  the  connector  reflect  the  same  pattern.  When  there  was  a  dominant  response, 
as  in  the  case  of  the  causals,  the  adversatives,  and  the  sequentials,  the  proportions  for  the 
ESL  and  the  native  English  speakers  were  almost  identical.  Second,  for  the  adversatives  and 
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the  sequential^  over  95%  of  the  justifications  were  accounted  for  by  connector-appropriate 
categories.  The  proportion  of  connector-appropriate  justifications  was  somewhat  lower  for 
the  additive  and  causal  connectors,  although  more  than  75%  were  in  connector- appropriate 
categories. 


Insert  Tables  17  &  18  about  here 


Justifications  for  Choices  that  were  Incorrect.  The  distribution  of  the 
justifications  for  incorrect  choices  are  shown  in  Table  18.  The  confusions  among  connectors 
that  were  reflected  in  the  contingency  analysis  of  the  errors  (Table  15)  were  generally 
reflected  in  the  justification  data.  When  students  were  justifying  incorrect  responses  that 
had  been  made  to  additive  slots,  the  majority  of  the  justifications  were  in  categories 
appropriate  to  causal  connectors.  This  is  consistent  with  the  data  showing  that  causal 
distractors  were  the  most  frequently  chosen  for  additive  slots  The  most  frequent 
justifications  for  incorrect  choices  for  causal  slots  were  in  categories  appropriate  for 
additive  connectors,  with  the  remaining  responses  distributed  over  the  causal-appropriate, 
adversative-appropriate  and  miscellaneous  categories.  The  additive  and  causal  distributions 
for  native  English  and  ESL  speakers  were  highly  similar. 

There  were  some  differences  between  the  language  groups  in  the  distributions  of 
justifications  for  incorrect  responses  made  to  adversative  and  sequential  slots.  For 
adversative  slots,  the  native  English  speakers  used  categories  appropriate  to  the  causal  56% 
of  the  time  and  categories  appropriate  to  the  additive  13%  of  the  time.  In  contrast,  the  ESL 
students  used  causal-  and  additive-appropriate  categories  equally  often  (34%  each).  The 
native  English  speakers'  justifications  of  incorrect  choices  for  sequential  slots  tended  to  be 
in  causal-appropriate  categories  most  frequently  and  additive-appropriate  or  sequential- 
appropriate  categories  equally  often  (24%  each).  On  the  other  hand  for  incorrect 
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sequential,  the  ESL  students  used  additive-appropriate  categories  most  frequently,  causal- 
appropriate  less  often,  and  rarely  used  sequential-appropriate  justifications. 

The  distractor  dependency  analysis  and  the  justification  data  considered  separately 
suggest  that  errors  may  be  due  to  a  "fuzzy"  understanding  of  certain  of  the  connectors.  To 
more  precisely  pursue  the  source(s)  of  difficulty  on  items  that  students  answered 
incorrectly  we  examined  the  relation  between  the  justification  and  the  connector  selected. 

Three  relations  are  informative  with  respect  to  source(s)  of  difficulty  and  these  are 
illustrated  in  Figure  13.  The  first  relation  -  incorrect  alternative  but  a  justification 
appropriate  to  the  alternative  chosen  -  indicates  that  the  error  is  due  to  difficulty 
processing  the  information  in  the  text:  the  student  has  selected  the  connector  that  matches 
the  relation  extracted  from  the  text  but  it  is  not  the  logical  relation  actually  called  for  by 
the  text.  For  example,  students'  understanding  of  the  text  may  have  made  them  think  that  a 
causal  connector  was  needed  in  an  additive  slot;  students  chose  the  causal  and  explained  their 
choice  using  a  causal-appropriate  justification.  Thus,  they  supplied  the  right  connector  for 
the  wrong  relation.  This  was  the  dominant  pattern  for  both  the  native  English  speakers 
(63%  of  56  opportunities)  and  the  ESL  speakers  (65%  of  122  opportunities). 

Insert  Figure  13  about  here 

The  second  type  of  relation  between  the  connector  selected  and  the  justification  - 
incorrect  alternative  but  with  a  justification  appropriate  to  the  cloze  slot  -  implies 
difficulty  with  the  meaning  of  the  connector  terms:  the  student  has  extracted  the  logical 
relation  called  for  by  the  text  but  does  not  choose  the  connector  that  conveys  that  relation. 

For  example,  the  student  explained  the  (incorrect)  choice  of  a  causal  saying  that  what  was 
needed  in  the  text  was  an  additive  connector.  This  pattern  did  not  account  for  many  of  the 
errors:  16%  for  the  native  English  speakers  and  6%  or  the  ESL.  The  third  relation 
reflects  problems  in  processing  the  text  and  in  matching  the  connector  terms  with  the 


inferred  relation.  In  this  case,  the  student  selected  an  incorrect  connector  and  provided  a 
justification  that  was  appropriate  to  neither  the  selected  connector  nor  the  cloze  slot  in  the 
text.  This  pattern  accounted  for  a  moderate  amount  of  the  native  English  speakers'  responses 
(21%)  and  for  a  somewhat  higher  percentage  of  the  ESL  students'  responses  (30%). 

Thus,  difficulty  extracting  the  appropriate  logical  relation  is  implicated  as  an 
important  source  of  student  errors:  inaccurate  inferences  about  the  appropriate  logical 
relation  between  information  contained  in  successive  sentences  in  the  text  accounted  for  the 
vast  majority  (95%)  of  the  ESL  students’  errors  and  84%  of  the  native  English  students' 
errors.  The  data  also  indicate  that  a  simple  "lack  of  knowledge"  of  the  functions  and 
meanings  of  various  connector-type  instances  was  not  a  primary  reason  for  incorrect 
responses. 

Discussion 

The  results  indicate  that  native  English  speakers  correctly  completed  more  of  the 
cloze  slots  than  did  the  ESL  students.  For  both  groups,  the  pattern  of  difficulty  among  the 
connector  types  was  similar:  cloze  slots  requiring  additive  and  causal  connectors  were  more 
likely  to  be  filled  in  correctly  than  were  cloze  slots  requiring  adversative  or  sequential 
slots.  This  pattern  of  differential  difficulty  partially  confirmed  our  expectations.  Although 
we  had  expected  sequentials  to  be  the  most  difficult,  we  had  not  expected  the  adversatives  to 
be  as  difficult  as  the  sequentials.  Justifications  for  correct  responses  were  similar  for  the 
two  language  groups  and  reflected  the  distinctions  among  connectors  that  we  had  postulated. 

When  incorrect  responses  were  made,  differences,  as  well  as  similarities,  emerged 
between  the  language  groups  and  among  the  connectors.  In  general,  when  native  English 
speakers  were  incorrect  they  completed  the  cloze  slots  with  causal  connectors  whereas  the 
general  tendency  for  the  ESL  students  was  to  choose  additive  connectors.  When  these  choices 
were  examined  contingent  on  the  correct  response  for  the  cloze  slots,  we  found  that  when 
additives  were  the  correct  choice,  both  language  groups  had  primarily  chosen  causals.  Both 
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groups  showed  similar  choices  when  {hey  incorrectly  responded  to  causal  and  adversative 
cloze  slots.  However,  the  two  groups  differed  on  their  incorrect  choices  for  sequential  cloze 
slots,  with  the  ESL  students  primarily  choosing  additives  while  the  native  English  students 
selected  both  causals  and  additives. 

In  both  the  ESL  and  native  English  speakers,  there  were  tendencies  toward  overuse 
of  the  additive  and  causal  connector  terms.  And  although  students  were  correct  most 
frequently  on  additive  and  causal  cloze  slots,  when  they  did  choose  adversative  and  sequential 
instances,  there  was  a  high  probability  that  these  choices  were  correct.  Taken  together, 
these  findings  suggested  the  possibility  that  students'  perceived  confidence  about  the 
appropriate  logical  relationship  and  choice  might  be  less  when  they  chose  additives  and 
causals  as  compared  to  when  they  chose  adversatives  and  sequentials.  We  pursued  this  issue 
in  Experiment  2  by  administering  a  confidence  rating  scale  along  with  the  forced-choice 
cloze  task. 

Experiment  2:  Effects  of  Connector  Types  on  Cloze  Completions  and 
Confidence  Ratings  of  Native  English  Speakers 

Experiment  2  tested  the  hypothesis  that  students  would  be  more  confident  of 
adversative  and  sequential  cloze  slot  completions  than  of  additive  and  causal  completions. 

This  study  also  served  as  a  replication  of  the  basic  findings  of  Experiment  1  with  a  new 
sample  of  native  English  speakers.  The  methods  used  were  similar  to  those  of  Experiment  1 , 
except  that  confidence  ratings  were  made  and  no  justification  data  were  collected. 

Method 

Subjects 

Thirty-two  native  English  speaking  undergraduates  enrolled  in  an  Introductory 
Psychology  class  at  a  university  in  Southern  California  participated  in  this  study  for  class 
credit.  The  demographic  data  reported  on  the  background  questionnaire  indicated  profiles 
similar  to  those  of  the  native  English  speakers  in  Experiment  1 . 


Design.  Materials  and  Procedure 


The  materials  and  design  were  identical  to  those  used  in  Experiment  1  with  the 
addition  of  a  confidence  scale  that  was  printed  on  every  page  of  the  response  booklet.  This 
scale  was  printed  in  number  line  form  and  ranged  from  1  {very  low  confidence)  to  7  (very 
high  confidence).  Beneath  each  number  was  printed  a  word  reflecting  the  degree  of 
confidence  represented  by  that  number.  The  procedure  was  similar  to  Experiment  1  except 
that  all  subjects  were  run  together  in  a  single  group  session  and  provided  confidence  ratings 
rather  than  justifications.  The  design  was  a  within-subjects  design  in  which  connector  type 
was  a  four-level  factor. 


Results 

Correct  Responses, 

As  in  Experiment  1 .  the  number  of  correct  responses  for  each  type  of  connector 
was  computed  for  each  subject.  The  results  of  a  one-way,  multivariate,  repeated  measures 
ANOVA  indicated  a  significant  effect  for  connector  type,  £  (3,  29)  =  8.86,  £  <  .001.  Four 
post  hoc  contrasts  of  the  means  shown  in  Table  19  were  computed,  using  the  Bonferroni 
procedure  (alpha  =  .0125).  As  in  Experiment  1,  additives  and  causals  were  correctly 
answered  more  frequently  than  were  adversatives  and  sequentials,  £  (1,  31)  =  13.4,  £  < 
.001,  Mserror  =  6.8.  Causals  were  correct  more  frequently  than  additives,  £  (1,  31)  = 
8.02,  £  =  .008,  Mserror  =  2.43.  The  difference  between  adversatives  and  sequentials  was 
not  significant,  £  (1,  31)  =  1.37.  Also  shown  in  Table  19  are  students’  confidence  ratings 
for  those  items  that  were  correct.  A  one-way  ANOVA  on  the  confidence  ratings  indicated  a 
significant  effect  of  connector  type,  £  (3,  29)  8.8,  a  <  .001.  Post  hoc  comparison  of  the 

additive  and  causal  ratings  with  the  adversative  and  sequential  ratings  confirmed  our 
prediction:  Confidence  ratings  were  higher  for  correctly  chosen  adversatives  and 
sequentials  than  for  additives  and  causals,  £  (1,  31)  =  23.75,  a  <  .001,  Mserror  =  771. 


Insert  Table  19 


Distractor  Choices. 

As  in  Experiment  1,  we  computed  the  distribution  of  each  subject's  total  errors 
over  the  distractor  alternatives.  These  proportions  were  transformed  using  the  arcsine 
transformation  and  submitted  to  a  one-way,  multivariate,  ANOVA.  There  was  a  significant 
effect  of  connector  type,  £  (3,  29)  =  14.65,  a  <  .001.  The  mean  proportions  and 
transformed  values  are  provided  in  Table  20.  Four  post  hoc  comparisons  (alpha  =  .0125) 
indicated  that  additives  and  causals  were  chosen  as  distractors  more  frequently  than 
adversatives  and  sequentials,  £  (1,  31)  =  44.85,  p  <  .001,  Mserror  =  -772.  Causal 
distractors  were  chosen  more  frequently  than  additives,  £  (1,  31)  =  7.57,  p  =  .01, 
Mserror  =  -605.  Adversative  and  sequential  distractors  were  chosen  equally  often. 
Confidence  in  these  incorrect  choices  was  not  significantly  different  across  the  four 
connector  types,  £  (3,  60)  =  2.16,  p  =  .1. 

Insert  Tables  20,  21  &  22  about  here 


The  results  of  analyzing  the  dependency  between  the  correct  connector  types  for  the 
cloze  slots  and  the  type  of  distractor  selected  when  errors  were  made  are  given  in  Table  21. 
The  pattern  replicates  the  pattern  observed  in  Experiment  1.  The  causal  connector  was  the 
dominant  distractor  for  additive  cloze  slots  and  the  distributions  for  the  adversative  and 
sequential  cloze  slots  were  virtually  identical  to  those  given  in  Table  15.  The  one  difference 
between  the  results  of  the  two  experiments  is  that  more  adversative  distractors  were  chosen 
in  causal  slots  by  the  students  in  the  second  study.  The  selection  rate  data,  shown  in  Table 
22,  indicated  a  bias  toward  causals  in  that  32%  of  the  choices  were  causals;  however,  only 
66%  of  those  choices  were  correct.  Adversatives  and  sequentials  were  chosen  least 


frequently  but  had  the  higher  percent  correct  rates.  Confidence  ratings  for  correct 
responses  were  highest  for  the  less  frequently  chosen  connector  types. 

Discussion 

The  results  of  Experiment  2  replicated  the  findings  of  Experiment  1  and  extended 
them  by  showing  that  students  were  more  confident  of  cloze  completions  for  adversative  and 
sequential  connectors.  Adversative  and  sequential  connectors  appear  to  have  more  restricted 
and  perhaps  clearer  usage  conditions  than  do  additives  and  causals.  Causals  and  additive 
;erms,  such  as  and  and  so  may  be  more  frequent  in  everyday  language  and  their  usage  may 
not  always  literally  connote  a  highly  restricted  meaning  compared  to  adversatives  and 
sequentials;  as  a  result,  some  of  their  more  specific  meanings  may  become  "diluted"  in  text 
more  so  than  the  meanings  of  adversatives  and  sequentials.  Students  may  choose  additives 
and  causals  because  they  are  more  familiar  in  everyday  speech,  not  because  they  are  certain 
about  their  appropriateness  in  the  contexts  of  written  text.  Hence,  even  when  their  choices 
are  correct,  students'  confidence  in  additive  and  causal  completions  is  weaker  than  their 
confidence  in  less  frequently  chosen  adversative  and  sequential  connectors. 

These  data  provide  support  for  the  interpretation  that  although  adversative  logical 
relations  and  sequential  temporal  relations  are  less  often  constructed  by  students,  when  they 
are  constructed,  students  are  confident  of  the  accuracy  of  their  inferences.  There  is 
significantly  more  doubt  when  causals  and  additives  are  selected  for  use  in  a  text. 

Experiment  3  pursued  these  issues  with  ESL  students  at  generally  lower  levels  of  English 
language  competency  than  the  individuals  who  participated  in  Experiment  1 .  Because  of 
this,  we  also  used  a  new  sample  of  texts  appropriate  to  this  population  and  a  somewhat 
different  set  of  connector  terms. 


Experiment  3:  Effects  of  Connector  Type  on  Cloze  Completions,  Confidence 
Ratings,  and  Response  Justifications  of  ESL  Students 

The  purpose  of  Experiment  3  was  to  examine  whether  ESL  students  confidence  in 
their  cloze  completions  varied  with  the  type  of  connector.  In  addition,  we  were  interested  in 
testing  the  replicability  of  the  ESL  results  from  Experiment  1 .  Due  to  constraints  on  the 
ESL  population  in  our  locale,  the  sample  for  Experiment  3  was  drawn  from  a  local  two- 
year,  community  college  rather  than  from  the  university.  Because  this  population  is 
generally  at  lower  levels  of  English  language  proficiency  than  the  university  population,  we 
developed  a  new  set  of  passages  based  on  texts  at  the  level  of  the  students'  proficiency  in 
English.  Thus,  Experiment  3  was  a  replication/extension  of  the  basic  findings  of 
Experiments  1  and  2  to  both  a  new  subject  population  and  different  materials.  Data  were 
collected  on  the  cloze  completion,  the  confidence  rating  and  the  response  justification  tasks. 

Method 

SublfiClS 

Participants  were  35  ESL  students  enrolled  in  the  advanced-level  ESL  English 
course  at  a  local  community  college.  Nineteen  of  the  students  were  enrolled  in  a  course  that 
met  during  the  day  and  16  attended  a  night  class.  All  students  in  the  class  participated  in  the 
paper  and  pencil  part  of  the  task  and  it  was  conducted  during  class  time.  Following  the  group 
session,  8  students  from  the  day  class  and  each  of  the  night  class  students  were  seen 
individually  for  purposes  of  explaining  their  response  choices  on  the  first  two  passages  that 
they  had  read. 

The  ESL  students  in  Experiment  3  were  more  heterogeneous  than  the  ESL  students 
who  participated  in  Experiment  1.  As  is  evident  from  the  data  in  Table  23,  the  subjects  in 
the  present  study  were  older  and  reflected  a  greater  age  range.  They  had  taken  fewer  related 
courses  and  the  level  of  English  language  skills  (M  =  5.3  grade  level  equivalent)  was  lower 
than  for  the  ESL  students  in  Experiment  1 ,  who  were  reading  at  levels  sufficient  to  enter  the 


university.  In  addition,  all  except  two  of  the  subjects  for  Experiment  3  had  started 
schooling  in  English  at  age  12  and  50%  had  actually  started  at  18  years  of  age  or  later. 


Insert  Table  23  about  here 


Design  and  Materials 

The  design  was  a  mixed  factorial  in  which  class  (day  or  evening)  was  a  between- 
subjects  factor  and  connector  type  (four  levels)  was  the  within-subjects  factor.  Dependent 
measures  were  derived  from  the  cloze  completion  responses,  the  confidence  rating  task  and 
the  justification  data.  Due  to  the  generally  lower  level  of  English  skill  in  this  population  of 
ESL  students  as  compared  to  the  university  students  of  Experiment  1 ,  new  passages  were 
developed.  The  same  constraints  on  the  occurrence  of  blanks  and  distribution  of  response 
alternatives  described  for  Experiment  1  and  2  were  followed  in  constructing  the  passages 
used  in  Experiment  3. 

Passages.  The  four  passages  were  modified  versions  of  texts  drawn  from  textbooks 
that  had  been  used  in  the  past  by  the  ESL  program  at  the  community  college  but  not  by  the 
particular  students  who  participated  in  the  study.  We  selected  passages  that  dealt  with  fields 
in  the  social  and  natural  sciences.  Specifically,  the  following  topics  were  discussed:  life 
styles  during  the  Paleolithic  Age,  emotions  that  cause  laughter,  the  natural  resources  in 
Siberia,  and  the  mystery  of  the  Bermuda  triangle.  After  modification  to  accommodate  the 
connectors,  the  mean  number  of  words  per  passage  was  518  and  the  mean  number  of 
sentences  per  passage  was  32.25.  For  three  of  the  passages,  the  Flesch  grade  level 
equivalent  was  9-10  and  for  1  it  was  7-8. 

A  passage  booklet  and  accompanying  response  booklet  were  created  for  each  subject. 
The  passages  were  presented  in  four  different  orders  to  counterbalance  position  effects  for 
the  experimental  passages.  The  practice  passage  was  always  the  first  one.  As  in  Experiment 
2,  each  page  of  the  response  booklet  contained  the  four  alternative  choices  for  a  particular 


cloze  slot  as  well  as  a  confidence  rating  scale.  The  same  constraints  described  for 
Experiments  1  and  2  on  the  ordering  of  the  response  alternatives  were  followed  in  this 
experiment. 


Procedure 

The  subjects  enrolled  in  the  day  class  completed  the  passage  and  response  booklets 
over  a  series  of  4  class  sessions,  each  session  lasting  for  55  minutes.  The  subjects  enrolled 
in  the  evening  class  completed  their  booklets  over  a  series  of  2  class  sessions  each 
approximately  2  hours  in  duration.  The  procedure  involving  the  booklets  was  identical  to 
the  one  used  in  Experiment  2.  After  completing  the  booklets,  a  sample  of  8  students  from 
the  day  class  were  seen  in  individual  sessions  and  asked  to  explain  their  response  choices  for 
the  first  two  passages.  Subjects  were  systematically  selected  so  that  a  discussion  of  each 
passage  occurred  an  equal  number  of  times.  All  subjects  enrolled  in  the  evening  class  were 
interviewed. 

Results 

Correct  Responses 

The  mean  probability  of  correctly  selecting  each  type  of  connector  was  computed 
for  each  student.  A  multivariate,  split-plot  ANOVA  revealed  no  main  effect  (£  (1,  33)  = 
2.93,  &  >  .05)  nor  interaction  (£  <  1)  involving  the  class  variable.  However,  consistent 
with  Experiments  1  and  2,  there  was  a  main  effect  of  connector  type,  £  (3,31)  =  13.76,  £<. 
001.  Five  post  hoc  comparisons  were  computed  using  the  means  shown  in  Table  24  and  the 
Bonferroni  procedure  was  used  to  control  the  Type  1  error  rate  (alpha  =  .01).  Additives 
and  causals  were  correctly  chosen  more  frequently  than  adversatives  and  sequentials,  £  (1, 
33)  =  36.27,  £  <  .001,  Mserror  =  -075.  Additives  and  causals  were  not  significantly 
different,  £  (1,  33)  =  3.47;  nor  were  adversatives  and  sequentials,  £  (1,  33)  =  3.45.  This 
pattern  replicates  the  results  obtained  in  Experiments  1  and  2:  Responses  for  additive  and 
causal  slots  were  more  often  correct  than  were  those  for  adversative  and  sequential  slots. 


In  contrast  to  the  findings  of  Experiment  2,  the  ANOVA  of  the  confidence  rating  data 
failed  to  reveal  any  significant  differences  due  to  connector  type,  £  (3,93)  =  2.19,  j*  =  .09, 
Mserror  =  -24.  Thus,  the  ESL  students  were  no  more  confident  of  their  adversative  or 
sequential  choices  than  they  were  of  their  additive  and  causal  choices.  Furthermore,  their 
confidence  ratings  for  their  correct  answers  were  about  1  scale  value  below  the  ratings  of 
the  native  English  speakers'  confidence  in  their  correct  answers. 

Distractor  Choices 

The  proportion  of  each  subject's  total  errors  associated  with  each  distractor 
alternative  was  transformed  using  the  arcsine  transformation  and  submitted  to  a 
multivariate,  split-plot  ANOVA  in  which  class  and  connector  type  were  factors.  As  in  the 
analysis  of  the  correct  responses,  there  was  no  effect  of  class,  £  (1,  33)  =  3.51,  p.  =  .07. 
There  was,  however,  a  significant  effect  of  connector  type,  £  (3,  31)  =  20.77,  p  <.001. 

The  means  are  shown  in  Table  25.  Five  post  hoc  comparisons,  using  the  Bonferroni 
procedure  (alpha  level  =  .01),  were  conducted.  Additive  and  causal  distractors  were 
selected  more  frequently  than  adversative  and  sequential  distractors,  £  (1,  33)  =  51.77,  p 
<  .001 ,  Mserror  =  -381 .  Causal  distractors  were  selected  significantly  more  often  than 
additives,  £  (1,  33)  =  13.23,  p  <  .001,  Ms  error  =  -24.  The  probabilities  of  choosing 
adversative  and  sequential  distractors  were  not  significantly  different  from  one  another;  nor 
were  the  probabilities  of  choosing  additive  and  adversative  distractors.  The  distractor 
choice  data  replicate  the  findings  for  the  native  English  speakers  in  Experiments  1  and  2  in 
that  causals  were  the  most  frequently  chosen  distractor.  For  the  ESL  students  in  Experiment 
1  there  was  a  tendency  to  choose  the  additives  and  causals  most  often;  thus  there  was  a  minor 
difference  between  the  two  ESL  samples. 


Insert  Table  25  about  here 
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ANOVA  on  the  confidence  ratings  for  the  incorrect  cloze  completions  revealed  a 
significant  effect  of  connector  type,  £  (3,  29)  =  3.96,  a  <  .01.  Post  hoc  contrasts  indicated 
that  confidence  was  higher  for  adversative  distractors  than  for  sequential  distractors,  £  (1 , 
31)  =  11.24,  a  =  .002.  Examination  of  the  patterns  of  the  means  shown  in  Tables  24  and 
25,  the  confidence  ratings  for  the  correct  responses,  suggested  that  there  was  a  potentially 
interesting  interaction  between  connector  type  and  ratings  for  correct  compared  to 
incorrect  choices.  The  ANOVA  on  these  data  indicated  higher  confidence  when  the  correct 
alternative  was  chosen  (M  =  4.93)  than  when  a  distractor  was  chosen  (M  =  4.57),  £  (1, 
28)  =  14.29,  a  =  .01.  There  was  also  a  significant  connector  type  by  correctness 
interaction,  £  (3,  26)  =  3.64,  a  =  -02.  The  interaction  was  pursued  with  three  post  hoc 
contrasts  (alpha  level  =  .017).  The  significant  contrasts  indicated  (1)  that  the  difference 
between  ratings  of  correct  and  incorrect  responses  was  larger  for  sequential  connectors 
than  it  was  for  the  other  three  connector  types,  £  (1,  28)  =  8.63,  a  =  -007,  Mserr0r  = 
8.57;  and  (2)  that  correct  selections  of  sequentials  received  higher  confidence  ratings  than 
when  sequential  distractors  were  chosen,  £  (1,  28)  =  17.33,  a  <  -001,  Mserror  =  -971. 

The  results  of  Experiments  1  and  2  suggested  that  both  native  English  and  ESL 
students  often  selected  causal  connectors  when  additives  were  most  appropriate.  In  the 
present  sample  of  ESL  students,  the  dependency  relation  analysis  indicated  a  much  weaker 
relationship.  As  the  data  in  Table  26  indicate,  50%  of  the  distractor  selections  for  additives 
were  causals  but  adversatives  were  selected  in  almost  40%  the  cases.  In  addition,  additive 
and  adversative  distractors  were  the  most  frequent  distractors  selected  when  errors  were 
made  on  causal  slots;  finally,  causals  were  the  most  frequently  selected  distractor  for 
adversative  and  sequential  slots.  It  appears  that  these  ESL  students  who  were  at  lower  levels 
of  English  language  proficiency  had  a  strong  tendency  to  overattribute  causality. 
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Insert  Table  26  &  27  about  here 


That  there  was  a  general  bias  toward  selecting  causal  connectors  is  supported  by 
the  analysis  of  the  connectors  selected  by  the  students.  The  data  are  provided  in  Table  27. 
First,  34%  of  the  choices  were  causals,  reflecting  the  tendency  to  choose  causal 
alternatives.  However,  the  likelihood  that  these  choices  would  be  correct  was  relatively 
low,  42%.  Sequentials  were  least  often  chosen  but  had  the  highest  likelihood  of  being 
correct.  Additives  were  given  the  highest  confidence  ratings,  in  contrast  to  the  pattern 
manifest  by  the  native  English  speakers  in  Experiment  2.  Thus,  although  the  adversative 
and  sequential  alternatives  were  selected  least  often,  they  did  not  manifest  the  pattern  of 
proportion  correct  and  confidence  ratings  that  prevailed  in  Experiment  2.  Nor  did  the 
present  sample  of  ESL  students  manifest  the  pattern  shown  by  the  ESL  students  in 
Experiment  1,  wherein  adversative  and  sequential  alternatives  were  selected  least 
frequently  but  were  most  frequently  correct.  It  is  possible  that  these  ESL  students  who  are 
at  less  sophisticated  levels  of  Enlgish  language  training  do  not  make  use  of  the  greater  degree 
of  constraint  governing  the  use  of  adversative  and  sequential  connectors.  The  verbal 
justifications  were  pursued  to  enlighten  the  criteria  that  these  students  were  in  fact  using. 
Justifications  otResponses 

Justifications  for  Correct  Responses.  Table  28  provides  the  proportion  of 
responses  in  each  of  the  justification  categories  for  each  of  the  connectors.  Several  trends 
are  important.  First,  the  majority  of  justifications  for  each  of  the  connectors  were  in 
categories  consistent  with  definitions  and  taxonomies  of  logical  connectors,  i.e.  60%  or  more 
of  the  responses  were  in  connector-appropriate  categories.  Furthermore,  the  dominant 
responses  within  each  category  were  consistent  with  the  distribution  obtained  from  the 
native  English  and  ESL  four-year  college  students  in  Experiment  1 .  Justifications  for 
additive  slots  were  divided  between  example  and  elaboration.  The  dominant  justification  for 


causal  slots  was  the  existence  of  a  cause-effect  relation;  the  dominant  justification  for 
adversative  slots  indicated  that  comparison,  contrast,  or  unexpected  information  was 
present.  For  the  sequential  slots,  justifications  involving  new  or  next  points  or  temporal 
relations  were  given.  There  was  some  difference  between  the  pattern  on  the  sequentials  in 
this  study  and  the  patterns  in  Experiment  1;  however,  in  designing  the  texts  for  Experiment 
3  we  purposely  tried  to  use  connectors  that  indicated  temporal  relations  or  "next"  points. 
Thus,  differences  in  the  justification  data  for  Experiments  1  and  3  are  undoubtedly  due  to 
the  specific  passages  and  connectors  we  included  in  the  materials. 

There  was  one  difference  between  Experiments  1  and  3  that  was  probably  not  due  to 
the  specific  passages:  Approximately  37%  of  the  justifications  for  correct  responses  were 
in  the  Miscellaneous  category  and  the  majority  of  these  were  choice  by  exclusion,  i.e.,  "The 
others  didn't  fit."  The  higher  frequency  of  the  exclusionary  category  suggests  that  the 
community  college  students  may  be  operating  with  a  greater  degree  of  implicit  or  tacit 
knowledge  about  these  connectors  than  are  the  university  students.  The  former  may  be 
trying  the  alternatives  to  see  which  "sounds"  best,  relying  on  implicit  knowledge  in  the 
process.  This  knowledge  may  be  unavailable  to  them  for  verbalization. 

Justifications  for  the  incorrect  responses.  As  with  the  justifications  for  correct 
responses,  about  30%  of  the  justifications  of  incorrect  responses  were  in  the  miscellaneous 
category,  with  no  difference  between  connectors  in  this  trend.  Consistent  with  the  results  of 
Experiment  1,  the  proportion  of  justifications  for  choices  that  were  incorrect  generally 
reflected  the  dependency  relations  that  were  reflected  in  Table  26.  For  example,  36%  of 
the  44  justifications  for  incorrect  causal-slot  responses  were  additive-appropriate  reasons 
and  18%  were  adversative-appropriate.  For  adversative  slots,  27%  were  additive- 
appropriate  and  27%  were  causal-appropriate  categories.  For  incorrectly  complete 
sequential  slots,  causal  -appropriate  explanations  were  provided  41%  of  the  time.  The 
additive  slots  where  the  only  ones  were  subjects  showed  more  than  a  slight  tendency  to  use 


connector-appropriate  connectors:  23%  of  the  justifications  for  incorrect  responses  were 
additive-appropriate. 
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The  justification  data  were  used  in  conjunction  with  the  completion  data  to  examine 
the  source(s)  of  difficulty  for  these  students.  Of  the  three  relations  discussed  in  the  context 
of  Experiment  1 ,  we  found  two  of  them  to  be  equally  likely  in  the  data  of  the  ESL  community 
college  students.  For  48%  of  the  justifications  for  incorrect  responses,  students  used  a 
justification  appropriate  to  the  distractor  they  had  selected  (branch  1  in  Figure  13).  This 
pattern  implies  difficulty  understanding  the  relation  called  for  by  the  text.  An  additional 
45%  of  the  justifications  for  incorrect  responses  used  a  justification  that  was 
inappropriate  given  the  slot  in  the  text  and  did  not  fit  the  distractor  selected  (branch  3  in 
Figure  13).  This  pattern  implies  difficulties  both  in  processing  the  relations  in  the  text  and 
in  understanding  the  meanings  of  specific  connector  terms.  Justifications  that  were 
inappropriate  to  the  choice  but  appropriate  to  the  slot  were  relatively  rare  (6%)  and 
indicated  that  knowledge  of  the  specific  connector  words  was  not  the  major  source  of 
difficulty.  Thus,  the  major  problem  seemed  to  be  in  processing  logical  relations  in  context 
and  recognizing  when  the  specific  types  of  logical  connectors  were  needed. 

Discussion 

The  patterns  of  results  in  this  study  were  generally  consistent  with  our 
predictions  and  the  results  of  those  of  Experiments  1  and  2.  Cloze  completion  responses 
were  more  frequently  correct  for  additive  and  causal  slots  than  for  adversative  and 
sequential  slots  and  justifications  for  the  correct  responses  reflected  the  appropriate 
connector  functions.  However,  contrary  to  expectations,  the  ESL  students'  confidence 
ratings  for  correct  choices  were  equivalent  to  one  another.  When  they  were  incorrect,  ESL 
students  most  frequently  chose  causal  distractors,  although  their  confidence  in  these  choices 
was  not  terribly  high.  The  confidence  rating  data  for  the  incorrect  choices  was 
unanticipated:  they  were  more  confident  of  wrong  answers  when  they  had  chosen 


adversatives  and  additives  than  when  they  had  chosen  causals  and  sequentials.  Finally,  the 
dependency  relations  between  the  correct  cloze  completion  and  the  type  of  connector 
incorrectly  selected,  in  conjunction  with  the  explanations  of  the  incorrect  choices  suggest 
that  in  this  sample,  the  ESL  students  had  a  less  precise  understanding  of  the  differences  in 
meaning  between  frequently  used  causal  expressions,  such  as  thus,  so,  and  hence  and 
frequently  confuse  them  with  simpler  additive  or  sequential  indicators. 

General  Discussion  of  Connector  Studies 

In  each  of  the  three  studies  there  was  a  consistent  effect  of  connector  type:  when 
cloze  slots  required  additive  or  causal  completion  terms,  students  were  more  likely  to  be 
correct  than  when  adversative  or  sequential  '•  >  were  needed.  We  had  predicted  that 
sequentials  would  be  difficult  because  (1)  there  are  a  greater  number  of  sequential  subtypes 
and  (2)  correctly  selecting  one  often  requires  a  reference  to  the  global  passage  rather  than 
to  the  local  clause  or  sentence  context.  We  were  somewhat  surprised  that  the  adversative 
tended  to  be  as  difficult  as  the  sequential.  We  suggest  three  plausible  explanations  of 
performance  on  adversatives.  The  specific  instances  of  the  adversative  may  have  lower 
frequency  of  use  than  the  instances  of  additives  and  causals.  Second,  performance  on  the 
adversative  may  be  affected  by  the  existence  of  a  reader  consistency  bias.  That  is,  readers 
may  be  operating  with  a  default  assumption  that  favors  interpreting  successive  sentences  as 
elaborating  on  the  old  material  rather  than  by  contradicting  it  or  restricting  its  scope. 
Finally,  recognition  of  an  adversative  relation  may  require  a  more  complex  backward  search 
to  prior  content  than  the  causal  or  the  additive. 

The  effect  of  linguistic  proficiency  on  correct  performance  was  consistent  with  our 
general  expectations.  ESL  speakers  were  able  to  correctly  complete  fewer  cloze  slots  than 
native  English  speakers;  community  college  ESL  speakers  performed  at  lower  levels  than 
university  ESL  speakers.  However,  it  is  important  that  the  connector  type  pattern  was 
generally  consistent  across  groups. 


That  there  was  a  response  bias  toward  the  additive  and  causal  instances  was 
illustrated  in  two  of  the  measures.  First,  causal  and  additive  connectors  were  the  most 
frequently  chosen  distractors.  Furthermore,  the  student  selection  rate  data  showed  a  strong 
tendency  to  choose  causals  and  additives.  The  selection  rate  analysis  also  showed  that 
adversative  and  sequentials  were  more  likely  to  be  correct  when  they  were  selected,  and 
students  had  the  highest  degrees  of  confidence  in  these,  although  this  effect  was  not  as  strong 
among  the  least  English  proficient. 

Analyses  of  the  incorrect  choices  revealed  some  interesting  information  regarding 
intersentential  reasoning  and  inference  making.  Causal  distractors  were  particularly  likely 
when  students  failed  to  choose  the  correct  additive  alternative.  Justifications  for  these 
errors  indicated  that  students  had  inferred  a  causal  relation  where  an  additive  had  been 
intended.  The  tendency  to  incorrectly  choose  a  causal  was  present  but  at  attenuated  levels 
for  the  adversative  and  sequential  slots.  The  strongest  trend  toward  choosing  the  causal  was 
present  in  the  data  of  the  community  college  ESI  students. 

The  patterns  associated  with  incorrect  responding  may  reflect  two  influences. 
Readers  may  be  using  the  causal  in  an  effort  to  create  relatively  tight  connections  among 
units  of  information.  Alternatively,  conversational  English  may  create  a  "sloppy"  meaning 
for  causal  connectors  such  as  so,  thus,  and  because.  That  is,  these  terms  may  be  used  in 
situations  where  a  relationship  other  than  cause-effect  is  being  discussed.  They  may  serve 
as  "psuedo  bridges"  rather  than  as  true  causals.  As  a  result,  greatest  overuse  of  the  causal 
would  be  expected  for  those  students  whose  dominant  experiences  with  English  have  been  in 
informal,  conversational  contexts.  Similar  confusions  in  understanding  the  causal  have  been 
reported  among  children  acquiring  English  as  a  first  language  (e.g.,  Corrigan,  1975). 
Content  domain  study  demands  more  precision  in  the  meaning  of  the  language  we  use; 
connectors  are  no  exception.  Thus,  as  students  engage  in  more  interaction  with  formal  text, 
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they  are  forced  to  refine  their  understandings  of  these  terms  and  the  usage  constraints  that 
govern  them. 

The  relationship  between  incorrect  responses  and  their  justifications  indicated 
that  errors  were  predominantly  due  to  incorrect  inferences  about  the  appropriate  logical 
relation  and  an  inability  to  find  the  connector  that  expressed  those  incorrectly  inferred 
relations  in  context.  When  students  had  inferred  the  appropriate  relation  between 
sentences,  they  rarely  erred  in  choosing  the  correct  connector. 

A  major  source  of  difficulty  for  students  doing  this  task  was  inferring  the 
appropriate  relation  between  successive  sentences.  For  informational  texts  such  as  these, 
designed  with  the  explicit  purpose  of  communicating  new  information  to  students,  it  seems 
particularly  important  to  use  logical  connectors,  especially  when  contrastive  points  are 
being  made.  It  is  also  clear  that  in  content  domains  where  it  is  important  for  students  to 
clearly  distinguish  between  psuedo  and  true  cause-effect  relations,  explicit  connectors  in 
the  text  will  facilitate  accurate  understanding.  We  also  want  to  emphasize  that  differences 
in  performance  between  native  English  and  ESL  students  were  largely  in  overall  levels  of 
performance  and  were  not  primarily  associated  with  differential  patterns  among  the 
connectors.  General  content-domain  comprehension  skills,  rather  than  specific  connector 
skills,  are  therefore  implicated  as  the  locus  of  the  language  group  differences.  Once  ESL 
students  have  grasped  the  basic  meaning  and  functions  of  instances  of  specific  connectors, 
further  drills  on  isolated  use  of  connectors  are  not  likely  to  lead  to  improved  performance 
on  connectors  in  natural  text  contexts.  Rather,  improving  ESL  proficiency  at  this  level 
seems  to  require  instruction  that  fosters  understanding  logical  relationships  between 
sentences  and  how  connectors  signal  such  relations  when  those  sentences  occur  in 
meaningful,  content-domain  contexts. 


The  research  conducted  under  the  auspices  of  this  project  generally  indicates  that 
there  are  more  similarities  than  differences  in  the  reasoning  and  comprehension  processes 
of  ESL  and  native  English  speakers.  There  were  few  differences  found  between  language 
groups  in  the  processes  applied  to  answering  questions,  to  reading  and  recalling 
information,  and  to  constructing  cohesion  between  sentences  of  a  lengthy  passage.  Linguistic 
characteristics  of  text  were  found  to  affect  those  processes  similarly,  regardless  of  language 
group.  Our  comparative  investigations  revealed  several  interesting  aspects  of  the  nature  of 
learner,  task,  and  text  interactions  in  text  processing  and  we  turn  to  a  discussion  of  these 
here. 

In  the  oceanography  protocol  study,  differences  between  more  and  less  successful 
learners  were  associated  with  domain-specific  background  knowledge  rather  than  with 
English  language  proficiency.  Those  with  greater  knowledge  of  domain-related  concepts 
performed  better  than  those  with  a  lesser  knowledge  base.  We  found  that  for  both  language 
groups,  certain  types  of  questions  were  more  aifncult  than  others.  The  more  difficult 
questions  were  those  that  required  application  of  information  from  the  text  to  new  situations 
and  those  that  paraphrased  statements  in  the  text.  The  one  difference  we  found  between 
native  English  speakers  and  ESL  speakers  was  that  the  latter  tended  to  engage  in  more 
extensive  text  searches  for  purposes  of  both  locating  question-relevant  information  and  for 
confirming  answers.  The  ESL  students'  greater  reliance  on  text  searches  emphasizes  the 
importance  of  being  able  to  thoroughly  decode  and  process  the  linguistic  form.  But 
weaknesses  in  processing  the  text  do  not  necessarily  impair  question  answering:  ESL 
students  with  high  levels  of  domain-specific  knowledge  may  compensate  for  less  than  perfect 
proficiency  in  English  by  activating  knowledge  of  principles  and  facts  relevant  to  questions, 
based  on  contextual  cues  discerned  in  the  text  and  questions. 

Finally,  the  protocols  also  revealed  an  interesting  "impasse”  that  occurred  when 
questions  had  a  high  number  of  words  that  matched  the  text:  Both  ESL  and  native  English 


students  often  engaged  in  lengthy  text  searches,  as  if  they  believed  that  the  answer  "was 
there  somewhere."  Particularly  for  questions  requiring  application  or  integration  the 
simple  search  strategy  did  not  produce  the  correct  answer.  Having  engaged  in  a  lengthy  but 
unsuccessful  search  for  the  answer,  students  frequently  gave  up  and  did  not  answer  the 
question.  They  rarely  re-evaluated  their  approach.  One  implication  of  these  impasses  is 
that  application  questions  should  be  indicated  as  such.  It  would  undoubtedly  facilitate 
learning,  especially  of  ESL  students  whose  dominant  solution  strategy  is  text  search,  to  more 
clearly  differentiate  between  questions  that  are  answered  "in  the  text"  and  those  that  require 
thinking  "beyond"  the  text. 

The  oceanography  protocols  of  both  native  English  and  ESL  students  indicated  that 
they  did  not  always  capitalize  on  the  linguistic  devices  that  cue  the  organization  of  discourse. 
Our  experimental  studies  pursued  one  class  of  such  devices,  connectors.  As  a  group, 
connectors  signal  the  logical  relationships  among  clauses  and  sentences  and  they  can  be 
important  cues  to  local  and  global  discourse  organization.  The  experimental  studies 
indicated  that  native  English  and  ESL  speakers'  ability  to  recall  information  was  aided  by  the 
presence  of  sequential  connectors.  However,  reading  behavior  appeared  to  be  governed  by  a 
broader  range  of  factors  than  the  simple  presence  of  a  sequential  marker.  A  broad  range  of 
semantic  and  structural  factors  appear  to  operate  in  concert  as  readers  proceed  through 
text.  It  will  be  important  in  subsequent  research  to  examine  these  factors  more  precisely. 
The  methodology  developed  in  the  present  context  is  an  important  tool  for  pursuing  these 
issues.  The  reading  strategy  analyses  indicated  that  individuals  used  a  mixture  of  strategies 
for  reading  text.  ESL  and  native  English  speakers  portrayed  similar  mixtures  of  strategies, 
although  there  was  a  tendency  for  the  ESL  students  to  review  text  materials  more  frequently 
than  did  the  native  English  speakers.  The  more  extensive  processing  of  t  ext  in  these  studies 
replicates  the  similar  finding  in  the  oceanography  study. 
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In  our  studies  of  four  types  of  connectors,  we  again  found  that  the  general 
orientation  to  the  task  was  similar  for  the  native  English  and  ESL  students.  Patterns  of 
correct  responding  as  well  as  of  incorrect  responding  were  consistent  across  groups.  In  all 
the  language  groups,  but  especially  in  the  least  English  proficient  ESL  students,  there  was  a 
tendency  to  overuse  causal  connectors.  This  tendency  may  reflect  the  influence  of  usage 
patterns  in  spoken  English,  where  speakers  often  use  causal  terms  even  when  a  cause-effect 
relation  is  not  intended.  Errors  were  associated  most  frequently  with  inappropriate 
inferences  about  the  logical  relation  between  sentences;  errors  were  not  primarily  due  to 
incorrect  understanding  of  specific  connector  terms  themselves.  One  implication  of  this 
research  is  that  the  inclusion  of  connecfors  to  signal  intended  intersentential  relations  is 
likely  (O  facilitate  appropriate  understanding.  The  clear  and  explicit  marking  of  the 
relation  in  this  way  is  likely  to  benefit  most  those  students  for  whom  English  language 
reasoning  and  comprehension  are  weanest. 

ESL  students  appear  to  apply  reasoning  and  comprehension  strategies  that  are 
very  much  like  those  used  by  native  English  speakers.  A  range  of  text  and  task 
characteristics  have  been  shown  to  similarly  affect  each  language  group.  Despite  the 
similarities,  the  ESL  speakers  were  less  efficient  than  the  native  English  speakers.  The  ESL 
speakers  took  longer  to  complete  every  one  of  our  tasks,  even  though  they  generally 
performed  at  lower  levels.  The  costs  associated  with  more  laborious  and  effortful 
processing  on  the  part  of  ESL  speakers  may  affect  motivation  and  general  attitudes  toward 
reading  and  schooling  as  well  as  the  efficiency  with  which  cognitive  operations  are 
performed  .  Motivational  costs  may  manifest  themselves  in  a  reduction  in  perseverance 
when  faced  with  assignments  deemed  difficult  even  for  native  English  speakers. 

Furthermore,  because  ESL  speakers,  even  at  the  university  level,  require  more  time  per 
assignment  than  native  English  speakers,  there  are  potential  effects  on  the  amount  of  work 
that  can  be  accomplished  in  a  fixed  period  of  time  and  on  the  number  of  courses  that  can  be 


8  1 

taken  in  any  given  school  term.  To  enhance  academic  and  economic  performance  of  ESL 
speakers,  important  questions  for  subsequent  research  are  those  that  inform  our 
understanding  of  (1)  cognitive,  motivational  and  attitudinal  costs  of  inefficient  processing 
and  (2)  the  development  of  efficient  reasoning  and  comprehension  in  English  ESL  speakers. 
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Table  1 


Question  types  and  distribution  of  experimental  questions  across  types 

1.  Verbatim  relationship  between  the  question  and  the  text. 


2,  3A:  "What  are..."  properties  of  concept 

12,  1A:  "How  many..."  quantitative  response 

12,  3A:  "What  fraction..."  quantitative  response 

12,  3C  &  D:  "What  (%)...”  quantitative  response 

12,  3E:  "Does  the  difference  (between  quantities)...” 


2.  Paraphrase  relationship  between  the  question  and  the  text,  including 
the  necessity  for  vocabulary  conversion  and  equivalence. 

2,  1  A:  "How  do  we..." 

2,  IB:  "Why  ..." 

2,5:  "How  can..." 

12,  2A:  "What  is..."  quantitative  response 


3.  Verbatim  "look  up"  plus  comparison. 


12,  IB  &  C:  "How  does  this  compare..." 
12,  2B:  "How  does  this  compare..." 

12,  5:  "How  are. ..similar?" 


quantitative  comparison 
quantitative  comparison 
concept  comparison 


4. 


Cross  paragraph  integration. 

2,  3B:  "Explain  (properties)..." 

2,  6:  "Explain  how..." 

12,  4:  "What  is  the  difference  between..." 
12,6:  "Explain  why..." 


concepts 

process 

concept  comparison 
concept  and  process. 


5.  Reasoning,  application,  computation  questions. 


2,  2:  "How  much..." 
12,  3B:  "What..." 


quantitative  response.  Formula:  not  all  the  necessary 
information  is  explicitly  in  the  text, 
quantitative  response.  The  next  value  in  a  series  must 
be  determined  from  the  series.  Alternatively,  the  rule 
that  generates  the  values  in  the  series  is  given  in  the 
text  and  may  be  applied  (see  12,  3E)  to  extend  the 
series. 
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Table  2 

Subject  characteristics 

Self-rating 

learning  from  English  command  of 


Learner 

Major 

SAT  (V/M) 

Native  Language 

lecture 

text 

science  vocab 

HSa 

Geology 

NA 

English 

Extr.  good 

Extr.  good 

Extr.  good 

MRb 

Physics 

560/650 

English 

Good 

Extr.  good 

Good 

LH 

Communications 

990-both 

English 

Extr.  c,jod 

Extr.  good 

Good 

DWC 

Sociology 

550/590 

English 

Extr.  good 

Extr.  good 

Good 

II 

Bus./econ 

380/540 

Croatian 

Good 

Good 

Moderate 

EH 

Bus./econ. 

550/500 

Dutch 

Good 

Extr.  good 

Moderate 

gl 

History 

520/440 

Spanish 

Extr.  good 

Good 

Moderate 

aThe  teaching  assistant  for  the  course  and  a  graduate  student  studying  Oceanography. 
bMR's  background  is  heavily  oriented  to  the  hard  sciences.  He  had  already  taken  two  other  College 
level  geology  courses.  In  combination  with  his  physics  major,  this  background  qualifies  MR  as  a 
subject  matter  expert  or  near  expert  in  the  introductory  course  in  Oceanography  because  many  of 
the  concepts  and  re.ationships  were  familiar  to  him  from  his  other  science  courses. 

CDW  had  had  one  year  of  foreign  study  in  Chile  and  lists  Spanish,  French  and  Russian  as  other 
languages  that  she  has  knowledge  of . 
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Table  3 

Question  Answering  Model  for  Learning  from  Academic  Texts 


Processing  Events 

Goals 

Encode  Question 

Goal: 

Determine  the  type  of  answer 
required  by  the  question. 

Goal: 

Determine  starting  point(s)  for 

searching  for  an  answer. 

Search  Memory  for  Answer 

Goal: 

Find  a  candidate  answer  that 

exceeds  criterion  for  response. 

(Evaluate  likelihood  of  success 

with  continued  search.  If  high, 

continue  memory  search;  If 
low,  try  another  means.) 

Goal: 

Determine  alternate  sources  for 

answer.  (Memory  search  may 

provide  information  that 

facilitates  external  search.) 

Search  External  Source  for 

Goal: 

Delimit  search  space.  (Use 

Answer,  e.g.  Textbook 

question  and  results  of  any 
memory  search.) 

Goal: 

Find  relevant  information. 

Goal: 

Process  the  text  information 

in  the  context  of  the  task 

defined  by  the  question. 

Construct  and  Output  Answer  Goal:  Answer  question  completely. 

Goal:  Match  type  of  answer  to  type  of 


question. 

Metacognitive  Processing, 
e.g.,  Confirm  Answer 

Monitor  quality  of  answer 
Self-Monitor 


Natural  language  protocol  for  DW  a 


1 .  Explain  why  you  feel  cold  when  you  get  out  of  the  shower. 

(Reading  the  question) 

2.  I  think  it's  coming  back  to  latent  heat. 

3.  Explain  why  you  feel  cold  when  you  get  out  of  the  shower. 

(Rereading  the  question.) 

4.  Because  of  the  evaporat’on  process. 

(Initial  Answer) 

5.  But  what  is  it  that  causes  that? 

6 .  Back  to  the  book. 

7.  "Suppose  we  take  a  certain  amount  of  water  saturated  air 

8.  and  raise  its  temperature  by  one  degree  Celsius. 

9.  Some  of  the  added  heat  would  go  into  raising  the  temperature. 

10.  For  example,  because  the  water  is  saturated,  increase... 

11.  We  know  that  nearly  twice  as  much  added  heat  can  go  into  evaporating  water." 
(Lines  7  -  11:  Reading  S29-S32  in  section  B.2) 

12.  "Since  the  fas..." 

(Begins  to  read  Si 5  in  section  B.2) 

13.  I'm  thinking  in  my  mind  at  the  same  time  as  I'm  reading  this 

1 4 .  that  I  could  simply  say  because  the  shower  was  so  warm, 

15.  and  you'd  get  out  and  the  air  temperature's  not  as  warm  as  the  shower, 

16.  but  I  know  that  there’s  something  in  the  book  that  I’m  missing. 

17.  Urn,  "  29%  continents" 

18.  "remainder  stored  polar  caps..." 

19.  "atmosphere  loses..." 

20.  Oh,  here  we  go. 

21 .  "The  ocean  loses  water  but  gains  it  back  through  precipitation, runoff... 
“(Lines  17  -  21:  Skimming  from  beginning  of  Chapter  through  section  A. 2  ) 

22.  "Everyone  is  aware  that  the  water  in  an  open  pan  will  slowly  evaporate. 

23.  The  warmer  and  drier  the  air, 

24.  the  faster  the  water  will  disappear. 


(table  continues) 
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Table  4  (continued) 

25 .  The  amount  of  water  the  air  can  hold  depends  on  .... 

(Lines  22  -  25:  Reading  SI,  2 , 3  under  B.1) 

26.  "room  temperature..." 

27.  "several..." 

28.  "content..." 

(Lines  26  -  28:  Skimming  S4  -  S9  in  section  B.i) 

29.  I  guess,  just  to  answer  that  simply, 

30.  Well,  let  me  check. 

31 .  So  "Absolute  humidity  is  simply  the  percentage  of  air  that  is..." 

(Reading  S10  in  section  B.I) 

32.  "Then  the  absolute  humidity... 

33.  "If  air  at  100%  humidity  is  cooled  then  it  becomes  supersaturated 

34.  and  the  excess  moisture  precipitates. 

35.  The  dew  point  is  that  temperature  at  which  the  air  will  become  saturated. 

36.  This  is  the  temperature  at  which  the  air  will  be  saturated. 

37.  Notice  that  the  dew  point  is... " 

(Lines  32  -  37:  Skimming  through  Si 4  -  SI 7  in  section  B.I,  occasionally 
reading  a  clause  of  a  sentence  aloud.) 

38.  "It  takes  a  lot  of  heat  to  evaporate  water. 

39.  "We  have  seen  that  one  calorie  will  raise  the  temperature  of  one  gram  of 
water  by  one  degree  Celsius. 

40.  Therefore,  it  takes  100  calories  to  bring  one  gram  of  water  from  it's 

freezing  point  filing , 

41 .  but  an  additional  540  are  required  to  actually  evaporate  the  water 

42.  Thus,  a  pan  of  water  on  the  stove  readily  comes  to  boiling, 

43.  but  it  takes  a  long  time  to  boil  dry. 

44.  If  it  weren't  for  this  we'd  have  difficulty  boiling  an  egg  or  making  soup.” 

(Lines  38  -  44:  Reading  Si  -  S6  in  section  B.2) 

45.  Ah,  hah. ..Wait. 

46.  "Therefore,  it  takes  100  calories  to  bring  one  gram  of  water  from  it’s 
freezing  point  to  boiling, 

47.  but  an  additional  540  are  required  to  actually  evaporate  the  water. 

48.  Thus,  a  pan  of  water  on  the  stove  readily  comes  to  boiling, 

(table  continues) 


Table  4  (continued) 

49.  but  takes  a  long  time  to  boil  dry. 

50.  If  it  weren't  for  this,  we'd  have  difficulty  boiling  an  egg  or  making  soup." 
(Line  46  -  50:  Rereads  S3-  S6) 

51 .  "This  is  also  why  we  perspire. 

52.  The  heat  required  to  evaporate  the  water  comes  off  of  our  skins,  cooling  us 
off. 

(Lines  51  -  52:  Reading  S7,  S8  in  section  B.2  ( end  of  the  first  paragraph 
section  B.2]) 

53.  Well,  I  guess  I'll  do  it  in  sort  of  a  round-about  way. 

54.  First  of  all,  the  heat, 

55.  the  hot  water  of  the  shower... 

56.  you  get  out  of  the  shower,  and  the  water,  which  is  storing  the  heat  on  your 
body,  is... 

57.  you  usually  dry  it  off 

58.  and  the  evaporation,  urn 

59.  Actually,  that's  not  true 

60.  cause  if  it's  cold  outside, 

61 .  the  evaporation  is  a  slower  process. 

62.  "The  heat  required  to  evaporate  the  water  comes  from  our  skin." 
(Rereading  S8  in  section  B.2) 

63.  Well,  I  guess  I'd  say  evaporation  and  leave  it  at  that. 

(Note:  She  ended  up  with  the  answer  she  started  with,  having  read  and 
reread  various  sections  of  the  text.  Si  7,  Si 8  in  section  B.2  actually 
contains  the  answer.) 

a  DW’s  statements  are  in  regular  type  face.  Material  that  is  being  read  from  the  text  is  in 
quotes.  Italicized  information  is  commentary. 


Table  5 


Example  of  a  coded  protocol  for  DW's  solution  to  question  12,6 


Cognitive  Action 

Information 

Comment/Interpretation 

Reads 

question 

Identifies 

topic  and  text  section 

"Latent  heat" 

Rereads 

question 

Recalls 

answer  from  memory 

Global  answer.  "Because 

of  the  evaporation  process" 

Questions 

self 

"But  what  is  it  that  causes  that?" 

Describes 

strategy  =  go  to  book 

Reads 

text  B.2  S29-S32 

Reads 

text  B.2S15 

States 

answer  (partially 

"Because  the  shower  was  so  warm 

correct  answer) 

and  you'd  get  out  and  the  air 

temperature’s  not  as  warm  as 

the  shower  but..." 

Monitors/Evaluates 

answer 

"1  know  that  there’s  some  thing  in 

the  book  that  I'm  missing.” 

Skims 

text  from  beginning  of  chapter 
through  section  A.2 

Reads 

text  section  B.1S1-  S3 

Skims 

text  section  B.1S4-9 

Describes 

next  action 

"1  guess,  just  to  answer  that 

simply..." 

States 

need  to  monitor 

"Well,  let  me  check" 

Reads 

text  section  B.IS10 

Skims 

text  section  B.1S11-17 

Reads 

text  section  B.2S1-  S6 

Recognizes 

relevant  information 

"Aha. ..Wait" 

Rereads 

text  B.2S3-S6 

Reads 

text  B.2S7-S8 

(table  continues) 
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Table  5  continued 

States 

Rereads 

Decides/States 


answer  that 
attempts  to  para¬ 
phrase  what  she's  read 
text  B.2S8 
answer  =  original 


Not  terribly  different  from 
initial  answer. 

Reverts  to  original  answer 


Table  6 

Strategies  for  correct  solutions  to  the  five  types  of  questions3 
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Percent  correct 

Type  1 

( n  =  3  5) 

89% 

Type  4 

(n-28) 

71% 

Question  type 

Type  3 
(n-21) 

62% 

Type  5 

(n=1 4) 

57% 

Type  2 

(n-28) 

39% 

Search  memory  (Type  A)  .32 

.45 

.54 

.50 

.25 

Search  text  (Type  B) 

.58 

.40 

.31 

.25 

.58 

Search  memory 

and  text  (Type  C) 

.10 

.15 

.15 

0 

.08 

Text  search,  reason 

beyond  (Type  D) 

0 

0 

0 

.25 

0 

No  added  processes 

.45 

.40 

.23 

.13 

.42 

Question  analysis 

.26 

.30 

.23 

.25 

.08 

Reasoning/Inference 

.13 

.05 

.54 

.63 

0 

Monitorinab 

AZ 

■  65 

A3. 

1.13 

Process 

.13 

.25 

.08 

.50 

.33 

Product 

.29 

.40 

.38 

.63 

.42 

aPercent  correct  solutions  is  given  in  parentheses  for  each  type  of  question.  Probabilities  in 
the  body  of  the  table  are  based  on  the  frequency  of  occurrence  of  each  event  in  correct 
solutions. 

bUnderlined  data  in  this  row  are  the  sums  of  the  probability  of  process  and  product 
monitoring.  Because  both  could  occur  in  a  protocol,  the  sums  can  exceed  1 .00. 


Table  7 

Solution  strategies  for  the  seven  learners3 
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Learners 


Native  English 

Nonnative  English 

MR 

HS 

DW 

LH 

GL 

BH 

II 

Number  Correct 

16 

12 

12 

8 

15 

10 

1  0 

Percent  Correct 

(89%) 

(6  7%) 

(6  7%) 

(44%) 

(83%) 

(55%) 

(55% 

Search  Type 

Memory 

9(8) 

10(7) 

5(4) 

10(5) 

3(2) 

3(3) 

7(4) 

Text 

5(4) 

7(5) 

7(4) 

7(3) 

15(13) 

1  0(5) 

9(5) 

Both 

3(3) 

-  -  - 

4(2) 

1  (0) 

.  .  - 

4(2) 

1(1) 

Compute 

1(1) 

1  (0) 

KD 

“  '  “ 

.  .  . 

1  (0) 

1  (0) 

Additional  Processing  Activities 

None 

7 

5 

2 

7 

1 

5 

6 

Question  analysis 

2 

2 

9 

8 

9 

6 

5 

Reasoning/inference  5 

6 

6 

5 

6 

4 

5 

Monitoring15 

a 

IQ 

11 

I 

11 

a 

IQ 

Process 

3 

8 

9 

3 

6 

8 

5 

Product 

8 

3 

9 

4 

6 

3 

6 

aMaximum  =  18  questions.  Entries  reflect  the  number  of  questions  whose  solutions  included 
a  particular  search  or  processing  activity  at  least  once.  Numbers  in  parentheses  indicate 
the  number  of  correctly  answered  questions  having  that  search  type  or  activity. 

^These  data  are  the  number  of  solutions  containing  either  process  or  product  monitoring. 
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Table  8 

Summary  profiles  of  the  seven  oceanography  learners 


• 

Successful  learners 

Knowledge 

Length  of 
solution 

MR  (89%  correct) 

Expert 

Short 

(67  actions) 

• 

GLa  (83%  correct) 

Moderately  successful  learners 

High 

Long 

(148  actions) 

HS  (67%  correct) 

Expert 

Short 

(92  actions) 

• 

DW  (67%  correct) 

Less  successful  learners 

Low 

Long 

(172  actions) 

• 

lla  (55%  correct) 

Medium 

Short 

(90  actions) 

EHa  (55%  correct) 

Low 

Short 

(83  actions) 

• 

LH  (44%  correct) 

aNonnative  English  speakers 

Low 

Short 

(80  actions) 

Monitoring 

Moderate 
(27  actions) 
High 

(46  actions) 

Moderate 
(31  actions) 
High 

(76  actions) 

Low 

(17  actions) 
Moderate 
(32  actions) 
Low 

(9  actions) 
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Table  10 

Mean  recall  of  the  target  points  (max  =  4)  by  Native  English  speakers  (Experiment  2),  ESL 
Speakers  (Experiment  3),  and  Native  English  Speakers  (Experiment  4) 

Signalling  condition 


Experiment  2 
Native  English 
Experiment  3 
ESL  speakers 
Experiment  4 
Native  English 


speakers  (n  =  32) 

(n  =  16) 

speakers  (n  =  16) 


Full 

2.5 


2.72 


3.13 


Number  Vague 

1.73  1.67 


2.44  1.84 


2.57  2.29 


None 


1 .55 


1 .72 


2.38 
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Table  11 

Mean  recall  of  the  signalled  and  unsignalled  target  sentences  in  Study  5a 

Target  Point 


PI 

P2 

P3 

P4 

No  points  marked 

Mono 

1.03 

.84 

1 .0 

.81 

(Number-Only  Condition) 

ESL2 

1.19 

1 .13 

1  .0 

1  .06 

• 

ESL3 

.56 

.63 

.56 

.69 

Signalled 

Mono 

1 .06 

1.13 

.92 

.91 

• 

ESL2 

.91 

1 .06 

.81 

1.19 

ESL3 

.56 

.56 

.53 

.69 

• 

Unsignalled 

Mono 

1.0 

.92 

.91 

.73 

ESL2 

1.13 

.57 

1  .0 

.78 

ESL3 

.56 

.81 

.56 

.50 

• 

a  The  maximum  score  in  each  of  the  cells  is  2. 
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Table  12 

Characteristics  ot  the  Students  in  Connector  Experiment  1a 


Characteristic 

Native  English 
Speakers 

ESL 

Speakers 

• 

Age 

Mean 

1  8.75 

20.3 

Range 

1  8-20 

1  8-30 

Age  at  which  schooling 

Mean 

-  - 

1 1 .63 

• 

in  English  began 

Range 

-  - 

4  -  26 

Scholastic  Aptitude  Test  (V)b 

Mean 

526.8 

338.3 

Range 

450  -  600 

210  - 

• 

Social  Science  Courses 

Mean 

2.78 

3.1 

in  High  School  or  College 

Range 

0  -  12 

0  -  15 

Major  Field  of  Study 

• 

Social  Science 

3  i  % 

30% 

Natural  Science,  including  Engineer 

60% 

Undecided 

s  r  % 

5% 

• 

Native  Language 

English 

1  00% 

0% 

Spanish 

5% 

• 

European  (except  Spanish) 

25% 

Asian 

70% 

a  All  data  are  self-reported.  We  did  not  have  access  to  official  student  records. 

Table  continues 
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Table  12  continued 
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b  All  16  Native  English  speakers  reported  Verbal  SAT  scores;  only  12  ESL  speakers 
reported  Verbal  SAT  scores.  The  mean  SAT  score  for  freshmen  at  this  university  has 
fluctuated  around  500  for  the  past  several  years.  An  additional  7  ESL  students  reported 
scores  on  the  Test  of  English  as  a  Foreign  Language  (TOEFL)  and  2  students  reported  both 
SAT  and  TOEFL  scores.  The  mean  TOEFL  was  564.7,  with  scores  ranging  from  510  to  630. 
The  mean  TOEFL  score  is  representative  of  that  reflected  in  the  nonnative  English  speaking 
undergraduate  population  at  this  university. 
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Table  13 

Mean  Number  Correct  on  Each  Type  of  Connector  in  Connector  Experiment  1a 


Language  Group 

Native  English 

ESL 

Overall  Connector 

Connector 

(n=  16) 

(n  =  20) 

Means  (n  =  36) 

Additive 

6.44 

5.7 

6.03 

Causal 

6.56 

5.0 

5.69 

Adversative 

6.19 

4.35 

5.17 

Sequential 

6.19 

4.45 

5.22 

Overall  Group  Means 

6.34 

4.88 

aThe  overall  means  have  been  weighted  to  reflect  the  unequal  sample  sizes.  The 
maximum  number  correct  was  8. 


Table  14 


Distractor  Selections  in  Connector  Experiment  1a 

Language  Group 


Native  English 

ESL 

(n=  16) 

(n  =  20) 

Mean  errors*3 

6.62 

12.20 

Distractor  Selected 

Additive 

.29 

.39 

Causal 

.41 

.31 

Adversative 

.05 

.14 

Sequential 

.16 

.13 

aThe  Anova  was  done  on  arcsine  transforms  of  the  proportion  of  incorrect  responses  for 
which  each  type  of  connector  was  selected.  The  means  of  the  transformed  measures  were  as 
follows.  For  the  native  English  speakers:  additive  =  1.1;  causal  =  1.38;  adversative  =  .44; 
sequential  =  .76.  For  the  ESL  speakers:  additive  =  1.37;  adversative  =  .73;  causal  =  1.16; 
sequential  =  .7. 

bThe  maximum  number  of  errors  was  32  (8  for  each  of  4  connectors). 


Table  15 


The  Proportion  of  Errors  on  Each  Connector  Type  Accounted  for  by  Each  of  the  Distractor 
Options  for  Connector  Experiment  1  .a 

Distractor  Options 


Correct  Connector 

Additive 

Causal 

Adversative 

Sequential 

Native  English  Speakers 

Additive 

.  .  . 

.76 

.08 

.16 

Causal 

.43 

*  .  . 

.09 

.47 

Adversative 

.40 

.43 

-  -  - 

.17 

Sequential 

.38 

.41 

.21 

-  -  - 

ESL  Speakers 

Additive 

-  -  - 

.60 

.24 

.15 

Causal 

.47 

-  -  - 

.15 

.38 

Adversative 

.49 

.40 

.  .  . 

.11 

Sequential 

.50 

.29 

.21 

-  -  - 

aThe  proportions  are  based  on  the  total  number  of  errors  for  each  connector  type.  For  the 
Native  English  speakers  the  number  of  errors  were  the  following:  On  additive  slots,  25:  on 
causal,  23;  on  adversative,  30;  on  the  sequential  slots,  29.  For  the  ESL  speakers  the 
number  of  errors  were  as  follows:  additive  ,  46;  causal  ,  60;  adversative  ,  72;  and 
sequential  ,  70. 


Table  16 


Student  Selection  Rate  Distributions  for  Connector  Experiment  1 


Percentage  of  Selections 

Percentage  of  Selections3  Scored  as  Correct 


Connector 

Native  English 

ESL 

Native  English 

ESL 

Additives 

27 

33 

76 

54 

Causals 

29 

28 

70 

56 

Adversative 

21 

1  9 

91 

71 

Sequential 

23 

20 

83 

70 

aTotal  number  of  selections  for  the  Native  English  students  was  512  and  the  total  for  the  ESL 


students  was  640. 
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Table  17 

•  Distribution  of  Justifications  for  Correct  Responses  for  Connector  Experiment  1a 


Connector  and  Justification  Category3  Native  English 

ESL 

Additives  -  Total  number  of  correct  responses 

46 

55 

Information  gives  example  of  concept 

.54 

.40 

Information  elaborates  prior  information 

.24 

.42 

Inappropriate  justification 

.17 

.15 

Causals  -  Total  number  of  correct  responses 

51 

46 

Cause-effect  relationship  between  two  sentences 

.71 

.67 

Consistent  but  vaguely  stated  logical  relationship 

Conclusion  about  a  cause  -  effect  relationship  that 

.08 

.04 

was  developed  over  several  sentences. 

.06 

0 

Inappropriate  justification 

.17 

.22 

Adversatives  -  Total  number  of  correct  responses 

Comparison  or  contrast  with  information  in  the 

45 

46 

prior  sentence;  unexpected  information 

.84 

.91 

Restriction  of  the  scope  of  the  prior  sentence 

.09 

.04 

Inappropriate  justification 

.04 

.02 

Sequential  -  Total  number  of  correct  responses 

44 

44 

Introduces  new  or  next  point 

.77 

.75 

Temporal  relation 

0 

0 

Sums  up  prior  or  pre',:  subsequent  information 

.20 

.23 

inappropriate  justir- n 

0 

0 

Miscellaneous  (exclusion,  •*  mg,  restating  text) 

.03 

.03 

aSee  Appendix  A  in  Goldman  and  Murray  (1989)  for  a  full  description  of  the  justification 
categories. 
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Table  19 


Mean  Number  Correct  on  Each  Type  of  Connector  and  Mean  Confidence  Ratings  for  Correct 
Responses  in  Connector  Experiment  2a 


Connector 

Mean 

Mean  Confidence 

Correct 

Rating  for  Correct 

Additive 

5.88 

5.54 

Causal 

6.66 

5.67 

Adversative 

5.63 

5.98 

Sequential 

5.22 

5.97 

aThe  means  are  based  on  32  subjects  and  the  maximum  score  was  8  for  correct 
responses.  Confidence  ratings  are  on  a  7-point  scale  with  1  =  very  low  confidence 
and  7  =  very  high  confidence. 


1  1  0 


Table  20 

Distractor  Selections  and  Mean  Confidence  Ratings  in  Connector  Experiment  2 


Distractor 

Proportion  of 

Mean  Confidence 

Incorrect  Responses3 

Rating  for  Incorrect13 

Additive 

.27 

5.04 

Causal 

.42 

5.39 

Adversative 

.13 

5.07 

Sequential 

.16 

4.97 

aThe  mean  errors  per  subject  was  8.62  out  of  a  maximum  of  32.  The  Anova  was  conducted 
on  the  arcsine  transformations  of  the  proportion  of  incorrect  responses.  The  means  of  the 
transformed  proportions  were  as  follows:  additive  =  1.06;  causal  =1.44;  adversative  = 
.68;  sequential  =  .77. 

bThe  means  for  the  confidence  ratings  were  based  on  a 
confidence  and  7  =  very  high  confidence.  The  means  are 
not  have  data  in  all  4  cells. 


7-point  scale  with  1  =  very  low 
based  on  21  subjects  because  1 1  did 


Table  21 


The  Proportion  of  Errors  on  Each  Connector  Type  Accounted  for  by  Each  of  the 
Distractor  Options  in  Connector  Experiment  2a 


Distractor  Options 

Correct  Connector 

Additive 

Causal 

Adversative 

Sequenti 

Additive 

-  -  - 

.60 

.13 

.26 

Causal 

.27 

.  .  . 

.27 

.44 

Adversative 

.46 

.42 

-  -  - 

.1 1 

Sequential 

.37 

.42 

.20 

-  -  - 

aThe  proportions  are  based  on  the  total  number  of  errors  for  each  connector  type.  The 
number  of  errors  were  the  following:  On  additive  slots,  68;  on  causal,  43;  on  adversative, 
76;  and  on  the  sequential,  89. 


Table  22 


Student  Selection  Rate  Distributions  for  Connector  Experiment  2 


Percentage 

Connector  of  Selections3 


Percentage  of  Selections 
Scored  as  Correct 


Mean  Confidence 
for  Correct  Responses 


Additives 

26 

70 

5.54 

Causals 

32 

66 

5.64 

Adversative 

21 

82 

5.96 

Sequential 

2  1 

79 

5.97 

aTotal  number  of  selections  was  1024. 
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Table  23 

Characteristics  of  the  ESL  Community  College  Students  in  Connector  Experiment  3a 


Characteristic 


Age 


Mean 

Range 

27.8 

19  - 

65 

20.6 

3  - 

60 

5.30 

2.7  - 

10.6 

2.34 

0  - 

28 

Age  at  which  school  ing  in  English  began 
Stanford  Diagnostic  Reading  Test,  Grade  Equivalent  5.30 
Social  Science  Courses  in  High  School  or  College13  2.34 
Native  Language 

Spanish  48.5% 

European  (except  Spanish)  8.5% 

Asian  43.0% 

aData  are  self-reported  except  for  the  Stanford  Diagnostic  Reading  Test  (SDRT)  Brown, 

Form  A.  The  SDRT  is  administered  by  the  community  college  for  placement  purposes  when 
students  enroll.  Scores  were  available  on  all  students.  Only  2  students  reported  a  TOEFL 
score 

bOnly  15  of  the  35  students  had  taken  courses  related  to  Social  Sciences  in  High  School  and 
only  9  had  taken  such  courses  in  community  college. 
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Table  24 

Mean  probability  of  a  correct  response,  mean  number  correct,  and  mean  confidence  rating 
for  correct  items  in  Connector  Experiment  3a 


Mean 

Mean  Number 

Mean  Confidence 

Connector 

Probability 

Correct 

Ratings  for  Correct*3 

Additive 

.61 

4.87 

5.1 1 

Causal 

.53 

4.72 

4.92 

Adversative 

.46 

3.76 

4.99 

Sequential 

.39 

2.89 

5.21 

aThe  maximum  number  correct  was  8  per  connector 

type,  except  for  the  sequential 

connector.  The  maximum  correct  for  the  sequential  connector  was  7.46  because  for 
1 9  of  the  students  one  of  the  sequential  slots  was  discounted  due  to  a  typographical 
error  in  the  passage.  The  error  was  corrected  for  the  other  16  students.  The  data 
for  probability  and  mean  number  correct  are  weighted  appropriately.  Due  to  this 
experimenter-error,  the  ANOVA  was  done  on  the  probability  correct  scores. 

^The  means  are  based  on  the  32  students  who  had  at  least  one  correct  response  for 
each  of  the  connectors.  Confidence  ratings  are  based  on  a  7-point  scale  with  1  =  very 
low  confidence  and  7  =  very  high  confidence. 
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Table  25 

Distractor  Selections  and  Mean  Confidence  Ratings  in  Connector  Experiment  3 


Distractor 


Proportion  of 
Incorrect  Responses3 


Mean  Confidence 
Rating  for  Incorrect*3 


Additive 

.25 

4.76 

Causal 

.39 

4.63 

• 

Adversative 

.20 

4.82 

Sequential 

.13 

4.50 

aThe  mean  errors  per  subject  was  15.77  out  of  a  maximum  of  32  The  Anova  was 
conducted  on  the  arcsine  transformations  of  the  proportion  of  incorrect  responses. 

The  means  of  the  transformed  proportions  were  as  follows:  additive  =  1.05;  causal 
=  1.36;  adversative  =  .91;  sequential  =  .72. 

^Three  subjects  did  not  have  data  in  all  4  cells  and  the  means  are  based  on  32  subjects. 
Confidence  ratings  were  based  on  a  7-point  scale  with  1  =  very  low  confidence  and  7  =  very 
high  confidence. 


Table  26 


The  Proportion  of  Errors  on  Each  Connector  Type  Accounted  for  by  Each  of  the  Distractor 
Options  for  Connector  Experiment  3a 


Correct  Connector 

Additive 

Distractor  ' 

Causal 

Options 

Adversative 

Sequential 

Additive 

— 

.50 

.39 

.10 

Causal 

.45 

.  .  . 

.37 

.17 

Adversative 

.28 

.44 

.  .  . 

.26 

Sequential 

.27 

.53 

.19 

-  -  - 

aThe  proportions  are  based  on  the  total  number  of  errors  for  each  connector  type.  The 
number  of  errors  were  the  following:  On  additive  slots,  108;  on  causal,  132;  on 
adversative.  152;  and  on  the  sequentials,  170.  The  number  of  possible  errors  was  280  for 
all  but  the  sequential  slots,  which  had  a  maximum  of  261 . 
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Table  27 

Student  Selection  Rate  Distributions  for  Connector  Experiment  3 


Connector 

Percentage 

of  Selections3 

Percentage  of  Selections 

Scored  as  Correct 

Mean  Confidence  for 

Correct  Responses 

Additives 

29 

53 

5.27 

Causals 

34 

42 

5.02 

Adversative 

23 

51 

5.09 

Sequential 

1  6 

57 

5.14 

aTotal  number  of  selections  was  1098,  representing  32  blanks  for  16  subjects  and  31  for 
19  subjects,  less  three  instances  of  no  response. 
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Table  28 


Distribution  of  Justifications  for  Correct  Responses  by  ESL  Community  College  Students  in 
Connector  Experiment  3 
Connector  and  Justification  Category 
Additives  -  Total  number  of  correct  responses 
Information  gives  example  of  concept 


61 

.44 


Information  elaborates  prior  information 

Inappropriate  justification 

Causals  -  Total  number  of  correct  responses 

Cause  -  effect  relationship  between  the  two  sentences 

Consistent  but  vaguely  stated  logical  relationship 

Conclusion  about  a  cause  -  effect  relationship  that  was 
developed  over  several  sentences. 

Inappropriate  justification 

Adversatives  -  Total  number  of  correct  responses 

Comparison  or  contrast  with  information  in  the 
prior  sentence;  unexpected  information 

Restriction  of  the  scope  of  the  prior  sentence 

Inappropriate  justification 

Sequentials  -  Total  number  of  correct  responses 

Introduces  new  or  next  point 

Temporal  relation 

Sums  up  prior  or  previews  subsequent  information 
Inappropriate  justification 
Miscellaneous3  (exclusion,  guessing,  restating  text) 


.31 

.11 

48 

.54 

.06 


0 

.15 

47 


.66 

0  • 

.19 


34 

.26  • 
.26 


.12 

.03  • 

.37 


Miscellaneous  did  not  differ  by  connector  type  and  the  .37  represents  the  rate  of 
this  response  over  all  connector  types. 
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Full  Number  Vague  None 

Condition 

Figure  1:  Mean  reading  rate  per  word  and  mean  processing 

time  in  four  signalling  conditions  for  ESL  speakers 


Seconds  per  word 
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Condition 

Figure  2:  Mean  reading  rate  per  word  and  mean  processing 
time  in  four  signalling  conditions  for  Native  English  speakers 
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PASSAGE  #13:  Pattern  Recognition  of  Speech  Signals 


Figure  3:  Once  Through  Strategy 
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PASSAGE  #8:  Location 


Figure  4.  Review  Strategy 
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PASSAGE  #16:  The  Nineteenth  Century 


Figure  5.  Regress  Strategy 
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Figure  6:  Proportion  of  Passages  Read  with  Each 

Type  of  Approach  by  Each  Language  Group 
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Figure  8:  Local  strategies  for  Review  Strategy  Passages 


Figure  9:  Local  strategies  for  Regress  Strategy  Passages,  Before  reaching  the  end 
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Figure  11: 
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Sentences  that  initiated  direction  changes  In  Regress  Strategy 
passages  before  reaching  the  end 
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Fig  12.  Sentences  read  to  strategically  in  Regress  Strategy  passages 
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Figure  13:  Relationships  between  incorrect  cloze  completions 
and  verbal  justifications 
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